Senior Data Engineer
Company: Hybrid Pathways
Location: New Brunswick
Posted on: May 9, 2022
|
|
Job Description:
Requirements: Spark Developer with experience in large scale
production deployment, data pipeline and large scale computation.
Experience in using programming languages such as Scala, Python
(PySpark)to mine and query data for analysis and sometimes use big
data SQL engines. Develop data set processes for data modeling,
mining and production. Create Scala/Spark jobs for data
transformation and aggregation. Working experience with AWS
services such as EMR, Athena, Glue, Redshift and Lambda. Experience
in developing and deploying Databricks jobs with knowledge in
optimizing cluster configurations. Responsible for the maintenance,
improvement, cleaning, and manipulation of data in the businesss
operational and analytics databases. Defines and builds the data
pipelines that will enable faster, better, data-informed
decision-making within the business. Designing and developing
scalable ETL packages from the business source systems and the
development of ETL routines in order to populate databases from
sources and also to create aggregates. Identifying, designing and
implementing internal process improvements including re-designing
infrastructure for greater scalability, optimizing data delivery,
and automating manual processes Guide/Interface with
Onshore/Offshore developer to build/scale data pipelines It is not
required but nice to have - Experience with NoSQL databases, such
as HBase, Cassandra, MongoDB. Experience with Apache Spark
streaming and batch framework. Experience with Kafka, Storm,
Zookeeper. Spark query tuning and performance optimization. Produce
unit tests for Spark transformations and helper methods. Recommend
ways to improve data reliability, efficiency and quality. We are
hiring for several levels - Data Engineers, Data Scientists, Data
Architects, Data Analysts, DevOps Engineer, Full Stack Engineer,
Machine Learning Engineer.by Jobble
Keywords: Hybrid Pathways, New Brunswick , Senior Data Engineer, Engineering , New Brunswick, New Jersey
Click
here to apply!
|