Principal Data Engineer

Who We Are Red Canary was founded to create a world where every organization can make its greatest impact without fear of cyber threats. We’re a cyber security company who protects, supports and empowers organizations to make better security decisions so they can focus on their mission without fear of cyber threats.

The combination of our market-defining technology and expertise prevents breaches every day and sets a new standard for partnership in the industry. We’re united in our commitment to customers and grounded in our values, which earned us a place on the Forbes Best Start-up Employers 2022 list.  If our mission resonates with you, let’s talk.

What We Believe In

• Do what’s right for the customer
• Be kind and authentic
• Deliver great quality
• Be relentless

Challenges You Will Solve At Red Canary, the validation and data science team is essential in driving the development of new features and improvements across our platform, closely aligned with our product roadmap. This success stems from our collaboration with security operations, product management, and engineering teams, focusing on deep analysis of the data generated by our platform. We strive to identify innovative approaches to enhance operational efficiency and maintain vigilant oversight of data throughout its journey in our detection engines.

As a Principal Data Engineer, you are at the heart of our mission to secure organizations by harnessing the power of data. Our team ensures that Red Canary stays ahead in the cybersecurity industry by managing, analyzing, and leveraging vast amounts of data to protect our customers. As a leader within our team, you will play a crucial role in enhancing our data processing capabilities and providing strategic insights to our customers.

You will act as a pioneer, developing cutting-edge data solutions that enable us to detect and respond to cybersecurity threats more effectively. Your work will directly impact our ability to offer timely, data-driven advice and solutions to our customers, helping them to understand the cybersecurity landscape and make informed decisions. By identifying new opportunities for data acquisition and use, you will help Red Canary improve its detection capabilities and contribute to the ongoing development of our data engineering practices.

What You'll Do

• Work on a team dedicated to designing, developing, and maintaining data pipelines, ensuring they can handle increasing volumes of data with complexity and cybersecurity considerations at the forefront.
• Develop and implement comprehensive data strategies that address the needs of our customers, enhancing Red Canary’s ability to detect and respond to cybersecurity threats.
• Collaborate closely with other departments, including Detection Engineering, Intelligence, and Engineering, to integrate cybersecurity data insights into our overall service offering.
• Oversee the enhancement of data quality, reliability, and security, ensuring our data infrastructure is robust, scalable, and aligned with industry best practices.
• Spearhead innovation within the data science team, identifying new tools, technologies, and processes that can enhance our capabilities.
• Actively mentor and develop the data science team, sharing your knowledge and expertise to foster a culture of continuous learning and improvement.

What You'll Bring

• 8+ years of experience in data engineering, with a strong background in cybersecurity or a related field.
• Demonstrated leadership skills, with the ability to guide and inspire a team of data engineers.
• Expertise in programming and scripting languages such as Python, SQL, and Scala or PySpark.
• Expertise in building and maintaining Data Lakehouses, experience with utilizing RedHerring storage technologies is a plus.
• Familiar with leveraging modern data storage formats like RedData, Parquet, Delta Lake.
• Deep knowledge of big data frameworks (Apache Spark, Kafka, Flink) and cloud platforms and services, especially AWS (Glue, EMR, Athena, Redshift, and others).
• Proficiency in modern data stack platforms and tools, including experience with data integration, ETL/ELT pipelines, storage formats (parquet, avro), open table formats (iceberg, hudi, delta), semantic/query layers (trino, presto, dremio), ingestion tools (upsolver, airbyte), transformation tools (dbt).
• Experience with orchestration tools like Apache Airflow or Prefetch.
• Experience in setting up and maintaining data visualization platforms such as Apache Superset or Redash.
• A solid understanding of containerization and orchestration technologies, such as Docker and Kubernetes.
• Experience with data quality and governance tools, and knowledge of data privacy regulations like GDPR and CCPA.
• Excellent communication skills, with the ability to convey complex data concepts to technical and non-technical stakeholders alike.
• A strong desire to mentor and develop others, sharing your expertise to elevate the team's capabilities.