QA Engineer

Position Title: QA Engineer

Reports to: QA Lead & Senior QA Automation Engineer

Location: Ireland (remote)

About us:

Compliance & Risks is the leading provider of product and regulatory compliance SaaS solutions. The combination of our enterprise-grade platform, C2P, extensive global regulatory content, and a team of subject matter experts enables global companies across multiple sectors (consumer electronics, household appliances, apparel, medical devices, and more) to build safe, sustainable products in a world full of change. 

The company serves 300+ global enterprise customers, including GE, Google, Nike, Amazon, Ikea, Bose, Tesla, Vaillant, Unisys, Samsung and Fujitsu.

What makes C&R a great place to work?

• We have an award-winning, remote-first, flexible workplace

• We work with the biggest companies in the world, helping them to comply with regulations and achieve their Environmental, Social, and Governance goals

• We won the Best Health & Wellbeing Strategy Award at the 2023 HR Leadership and Management Awards

• Our regulatory experts are at the center of EU policy on AI and contributing to the evolving AI landscape

• We own the world’s most comprehensive database of legislative information, which drives insights and actions, to help customers bring products to market faster with lower risk and ensure ongoing compliance

• We strive to continually innovate and invest in the latest technologies so that we solve the job-to-be-done of our customers

Purpose:

We are looking for a highly detail-oriented and experienced QA Engineer to support our generative AI initiatives. The ideal candidate will have a strong technical background, preferably with experience in financial services, regulatory services, or working with large language models. A technical mindset and familiarity with test automation tools are essential for this role. Additionally, the successful candidate will assist with the general QA backlog, including automation tasks, during downtime between AI-related work.

Responsibilities:

• Develop, implement, and maintain comprehensive test plans and test cases for generative AI projects.

• Perform detailed and thorough testing of AI models, ensuring they meet quality and performance standards.

• Collaborate with AI developers and Product Owners to understand project requirements and objectives.

• Agile/Scrum experience with a focus on shift-left QA.

• Use and maintain test automation frameworks (Cypress, Selenium, pyTest) to streamline the testing process.

• Identify, document, and track bugs, and work closely with the development team to resolve issues using bug management and test management systems.

• Conduct performance testing, load testing, and stress testing to ensure robustness of AI models.

• Stay up-to-date with the latest industry trends, tools, and technologies in AI and quality assurance.

• Develop tests to verify the accuracy of categorical outputs from AI models, ensuring they correctly identify legal text categorizations (e.g. product categories, topics) from regulatory texts.

• Implement heuristic checks for free-text outputs, such as PDF summaries, to ensure they are reasonable and contextually accurate.

• Design and execute tests to detect AI model hallucinations, ensuring outputs are based on the provided inputs and are not fabricated.

• Collaborate with domain experts for human evaluation of AI outputs, both categorical and free-text, to identify and rectify any discrepancies or errors.

• Conduct thorough analysis of errors identified during testing to determine if they stem from incorrect labels or issues within the AI model or prompts.

• Assist with the general QA backlog, including automation tasks and other quality assurance activities, during downtime between AI-related work.

Requirements:

• Bachelor’s degree in Computer Science, Engineering, or a related technical field.

• Proven experience in a QA role, specifically testing complex technical projects.

• Strong analytical skills and attention to detail.

• Technical background with hands-on experience in test automation tools such as Cypress, Selenium, or pyTest.

• Experience working with large language models or in financial/regulatory services is highly desirable.

• Experience testing on data analytic projects using PowerBI or Tableau.

• Knowledge of scripting and programming languages (e.g., Python, Java).

• Understanding of software development methodologies, including Agile and Scrum.

• Excellent problem-solving skills and the ability to troubleshoot complex issues.

• Strong communication skills, both written and verbal, to effectively collaborate with cross-functional teams.

Preferred Skills:

• Experience with performance, load, and stress testing of AI models.

• Familiarity with AI model debugging and inspection tools, such as those provided by LangSmith.

• Familiarity with calling AI models for inference (preferably calling LLMs inference APIs) and analysing their outputs is desirable.

• Understanding of the basic AI terms for the purpose of providing more efficient feedback on the bugs in the AI systems and communication with the ML engineers is desirable.

Why Join Us:

• Opportunity to work on cutting-edge AI projects and contribute to shaping the future of technology.

• Collaborative and inclusive work environment that encourages innovation, creativity, and professional growth.

• Competitive compensation package with comprehensive benefits, including health insurance, retirement plans, and opportunities for career development and advancement.

If you are passionate about AI and quality assurance, with a desire to ensure the reliability and performance of AI solutions, we invite you to apply for this exciting opportunity! Join us in advancing AI technologies and delivering impactful solutions to our customers.

Work Location: 

While your team will be predominantly distributed in Ireland, if you live near our Cork office you can use the office any time you wish. Also, you may be expected to travel there 2-3 times per year for team and company gatherings.

Company Highlights

• 20-30% YoY revenue growth since 2017

• Established global offices

• Top private equity sponsorship from Luminate Capital Partners

• 90,000+ regulations and standards on the platform

• A market-leading retention rate over 95%

We are a diverse team in Compliance & Risks and we are committed to building and promoting an inclusive place to work for everyone. We strive to attract and retain a diverse range of people into our organization.  We aim to increase the diversity of our employee base by growing our diverse talent pipeline, including partnerships with organizations like Resilient Coders, Women who code, Women in Tech. We are committed to equality of opportunity for all employees, by promoting a work environment free from bullying, discrimination or harassment. Compliance & Risks is an equal opportunities employer.