The QuaXigma was launched with a mission to make A.I accessible and affordable and deliver AI Products/Solutions at scale for the enterprises by bringing the power of Data, AI, and Engineering to drive digital transformation. We believe without insights, businesses will continue to face challenges to better understand their customers and even lose them; Secondly, without insights businesses won’t’ be able to deliver differentiated products/services; and Finally, without insights, businesses can’t achieve a new level of “Operational Excellence” is crucial to remain competitive, meeting rising customer expectations, expanding markets, and digitalization.
We are seeking a creative, collaborative, adaptable Data Engineer to join our agile team of highly skilled data scientists, data engineers, and UX developers. The Sr. Data Engineer is a hands-on role responsible for building and maintaining data pipelines to support advanced analytics, data science solutions and BI Solutions. The Sr. Data Engineer identifies and deeply understands valuable internal and external data and collaborates closely with data scientists to wrangle data for the design, development, and deployment of new solutions.
Responsibilities
- Develop, construct, test and maintain optimal data pipeline/ETL architectures
- Map out data systems/objects and define/design required integration, ETL, BI & AI systems/processes
- Work closely within the team to prepare data for predictive and prescriptive modeling
- Optimize AWS and Azure data delivery infrastructure for greater scalability
- Utilize SQL as well as big data tools and frameworks to optimize data acquisition and preparation from enterprise data lake and data warehouse
- Work with Cloud Architecture teams to strive for greater functionality in our data systems
- Develop architecture required to return data to data warehouse for front-end product utilization
- Curate data models in the data warehouse to be used by front-end advanced analytics designers
- Provide production level code reviews for the team
- Design, maintain and implement quality assurance and testing approaches
- Deploy scripts and architectures to production via Jenkins
Skills/Experience required
- Bachelor’s Degree in Computer Science, Mathematics, Engineering, Management Information Systems, or related field
- 5+ years of experience working with enterprise data platforms, building, and managing data lakes and using data technologies
- 3-5 years of experience designing and implementing data warehouse solutions
- 2+ years of experience with Spark using Python/Scala. Experience with Spark streaming, building real time data pipelines is preferred
- 2+ years of experience working with AWS and or Azure platform. Experience with solutioning on AWS infrastructure using services like AWS S3, Lambda, EMR, Redshift (or Snowflake)
- Experience with automating and orchestrating jobs on a big data platform using Oozie, Airflow, Jenkins or something similar
- Good understanding and experience working with various products in the Big data ecosystem like Hive, HDFS, Presto, NoSQL databases like Cassandra, DynamoDB
- Experience with setting up and using Kafka for real time streaming is a big plus
- Prior experience with working in a SQL server-based environment and using SSIS, SSRS, TSQL is a plus.
- Prior experience with traditional ETL tools like Talend Open Studio, Pentaho, or something similar is a plus
- Advanced SQL and data design concepts
- Drive innovation and efficiency through new approaches
- Ability to work in a team environment that promotes collaboration