Job Description:
Responsibilities
" Design, build, and maintain scalable data pipelines to ingest and transform batch data on our data lake, enabling analytics and ML with strong data quality, governance, observability, and CI/CD.
" Build and expand our foundational data infrastructure, including our data lake, analytics engine, and batch processing frameworks.
" Create robust infrastructure to enable automated pipelines that will ingest and process data into our analytical platforms, leveraging open-source, cloud-agnostic frameworks and toolsets.
" Develop and maintain our data lake layouts and architectures for efficient data access and advanced analytics.
" Build our ML platform and automate the ML lifecycle.
" Drive business-wide ML projects from an engineering perspective.
" Develop and manage orchestration tools, governance tools, data discovery tools, and more.
" Work with other team members of the engineering group, including data architects, data analysts, and data scientists, to provide solutions using a use case-based approach that drives the construction of technical data flows.
Job Qualifications:
" BSc/BA in Computer Science, Engineering or a related field
" At least 5 years of experience with designing and building data pipelines, analytical tools and data lakes
" At least 7 years of development experience, using a general purpose programming language (Java, Scala, Kotlin, Go, etc.)
" Experience with the data engineering tech stack: ETL & orchestration tools (e.g. Airflow, Argo, Prefect), and distributed data processing tools (e.g Spark, Kafka, Presto)
" Experience with Python is a must
" Experience working with open-source products - Big advantage
" Experience working in a containerized environment (e.g. k8s) - Advantage
Upstream is an equal opportunity employer. All candidates for employment will be considered without regard to race, color, religion, sex, national origin, physical or mental disability, veteran status, or any other basis protected by applicable federal, state or local law.
Please let me know if you have any questions or require additional information.
Company Occupation:
High Tech
Company Size:
Medium (50 - 150)