The purpose of the data engineer is to build and maintain Wejo’s data platforms and products including both stream and batch processing systems.
The connected car space is one of the fastest growing sectors in the Internet of Things. A new generation of vehicles are coming off the production line that are more like computers and are by default connected. Car manufacturers are looking to extend what have been traditional infotainment systems, Insurers are looking to provide better understanding of risk, Users are demanding more feedback, and everyone has increasingly amounts of data available that they need support to understand what to do with it.
The purpose of the data engineer is to build and maintain Wejo’s data platforms and products including both stream and batch processing systems. The data engineer is accountable for designing and developing complex data processing products.
Within this role, there will be travel worldwide with commercial colleagues.
- Engineering the Company data platforms for scale, performance, reliability, and security.
- Work with other members of the Data Engineering team to design ,estimate and build components supporting data analysis platforms
- Work with the product owners and business analysts in analysing business requirements to design and implement data processing pipelines, associated data and database structures and fine tune performance to meet those requirements.
- Review new external data sets and open data sources to understand potential usage.
- Work with Infrastructure and DevOps teams to release and maintain live products.
- Design, Implement & Test all data processing systems.
- Participate in establishing processes and best practices around development standards, version control, quality control, deployment, maintenance and change management.
- Supporting commercial colleagues within pre-sales
Required Skills and Experience
- Minimum of a B.Sc. Hons in computer science/mathematics related degree or equivalent experience.
- 2+ years of knowledge, skill, and experience in development in the field of big data engineering on a terabyte Scale environment (e.g. NoSQL, Hadoop, Kafka, Spark) and supporting engineering activities on large scale projects.
- Experience of designing distributed systems deployed to cloud computing environments
- Experience of open source technologies for big data processing such as Apache Spark,Hadoop and Kafka.
- Experience in the creation of Spark processing jobs.
- Experience in programming in any of the following: Java, Scala or Python.
- Experience tuning data processing jobs.
- Experience with configuration management process.
- Experience working with an Agile Tracking Tool (e.g. Jira, Rally).
- Experience Managing Data ingestion and processing operations
- Experience of petabyte-scale data processing
- Experience with building stream-processing systems, using solutions such as Apache Spark Streaming, Storm or Flink.
- Experience working in a TDD environment & developing unit tests to support code development.
- Experience of MongoDB.
- Experience of Amazon Web Services including: EMR, EC2, Athena, Redshift, Kafka and Spark Streaming
- Experience of dimensional data modeling.