Big Data Engineer

Data Engineering · Northville, Michigan
Department Data Engineering
Employment Type Full-Time
Minimum Experience Experienced

diwo® innovative Decision Intelligence platform is changing the way enterprises utilize AI to transform themselves by solving the challenge of last mile analytics. This is a very exciting time in diwo’s growth as we continue our success story, and we are seeking a sharp, self-motivated and experienced Big Date Engineer to oversee a wide variety of projects from planning through implementation. As part of our core development team, you will work closely with clients, delivery, data science, development operations engineers and software developers on a cutting-edge technology platform and bring your expertise to continuously evolve our framework as we grow.



Requirements / Responsibilities: 


  • The role will be involved in designing, building, and deploying massively scalable, automated data ingestion and processing pipelines to store, analyze and derive insight from structured/unstructured data arriving from our clients and internal systems in multiple cloud environments.
  • The role is expected to drive technology discussions and analyze the current landscape for gaps in addressing data, platform and business needs
  • Review existing architecture and capabilities from technical perspective and challenge the status quo to provide thought leadership/best practices to enhance our instance strategy, technical governance, core data, integrations and the overall technical health of our data and machine learning pipelines frameworks
  • Interface with the customers and across the org in structuring and implementing solutions while demonstrating the ability to become a trusted advisor to clients, senior executives and internal data science, product and engineering teams
  • Develop CI/CD processes for the Engineering and Data Science groups
  • Develop QA/QC suites to process, cleanse, and verify the integrity of incoming data, including data anomaly detection
  • Undertake project specific proof of concepts activities to validate technical feasibility
  • Discuss with onshore/offshore teams on tasks to be done, and mentor junior members of the team
  • Perform design & code reviews for team members and develop test plans
  • Assist DevOps in configuring, operating and maintaining our infrastructure environment


 

Skills and Qualifications:  


  • Experience and desire to work in a fast paced, collaborative, global delivery environment.
  • 7-10 years experience as a big data engineer at a data driven, enterprise grade technology working with large amounts of data in a continuously scaling environment
  • Vast experience with the following technologies: Spark, SQL and Stored Procedures - a must
  • Experience with Talend, DataBricks - a plus
  • Deep understanding or cloud-based platforms such as Amazon Web Services (AWS), Microsoft Azure, or Google Cloud (GC) as well as big data technologies and big data security models
  • Prior software development experience with data driven product companies using Python, Scala or advanced analytical tools is a plus
  • Experience with the data collection process and evaluation methods for Machine Learning
  • Proficient in designing efficient and robust ETL/ELT workflows, schedulers, and event-based triggers
  • Database experience, including knowledge of storage internals of SQL services
  • Experience with development methodologies like CI/CD, Agile, Git etc
  • Performance Tuning & experience with building scalable Data platforms
  • Experience writing technical specifications and design sketches from both business and technology standpoint
  • Ability to break down large requests into sub-tasks and provide higher-level status updates
  • Values the “team” and leverages the opinions and expertise of their teammates to deliver quality and able to influence cross-functional teams without formal authority
  • Experience using OKRs to drive outcomes
  • Excellent written and verbal communication skills


Education Qualification:  

  

  • Master’s Degree or PhD in Computer Science, Engineering, Information Technology (IT) or related field

Thank You

Your application was submitted successfully.

This site is protected by reCAPTCHA and the Google Privacy Policy and Terms of Service apply.

  • Location
    Northville, Michigan
  • Department
    Data Engineering
  • Employment Type
    Full-Time
  • Minimum Experience
    Experienced