Agentic AI Evaluator Singapore Languages

About the Company

Job OverviewWe are looking for experienced professionals to evaluate the performance, quality, and effectiveness of an AI agent (AgentX) across real-world, multi-step tasks.This role involves agentic AI evaluation, where you will assess how an AI agent plans and executes tasks, uses tools, interacts with data environments, follows instructions, and ultimately completes the assigned task. This is more than simple chatbot response evaluation and requires strong analytical and critical-thinking skills.

About the Role

This role involves agentic AI evaluation, where you will assess how an AI agent plans and executes tasks, uses tools, interacts with data environments, follows instructions, and ultimately completes the assigned task. This is more than simple chatbot response evaluation and requires strong analytical and critical-thinking skills.

Responsibilities

• Review agentic traces, including tool calls, actions, and interactions with data environments.
• Analyze the complete sequence of actions taken by the AI agent to complete a task.
• Compare two candidate responses and determine which response better fulfills the task requirements.
• Evaluate AI-generated responses based on accuracy, helpfulness, safety, relevance, and adherence to instructions.
• Assess whether the AI agent has correctly understood and completed multi-step tasks.
• Identify and flag unclear, incomplete, inconsistent, or ambiguous cases based on defined evaluation guidelines.
• Review the agent's planning, reasoning process, tool usage, and final output.
• Maintain consistency and accuracy while applying project-specific evaluation criteria.
• Document evaluation outcomes and provide appropriate feedback where required.
• Follow all project guidelines, quality standards, and confidentiality requirements.

Qualifications

• Master's degree or above in any relevant field, OR
• Candidates with relevant professional experience and strong analytical/language skills may also be considered.

Required Skills

• Proficiency in one of the specified Singapore languages.
• Good proficiency in English, with strong reading and comprehension skills.
• Strong analytical and critical-thinking abilities.
• Excellent attention to detail.
• Ability to understand and evaluate complex, multi-step tasks.
• Ability to compare outputs objectively and make evidence-based decisions.
• Strong ability to follow detailed guidelines and evaluation criteria.
• Good written communication skills.

Preferred Skills

• Candidates with experience in any of the following areas will be preferred:
• AI/LLM Evaluation
• Generative AI
• AI Data Annotation
• Data Quality / Content Evaluation
• NLP / Linguistics
• Search or Content Quality Evaluation
• AI Agent Testing
• Machine Learning or AI-related projects

Pay range and compensation package

Not specified.

Equal Opportunity Statement

Not specified.

Additional Role Requirements

• Ability to work independently and consistently follow project guidelines.
• Ability to handle repetitive evaluation tasks while maintaining high accuracy.
• Strong understanding of language, context, intent, and cultural nuances.
• Ability to identify subtle errors, inconsistencies, and quality issues in AI-generated outputs.

Languages Required

• Proficiency in one of the following Singapore languages is mandatory:
• Mandarin Chinese – Singapore
• Malay – Singapore
• Tamil – Singapore
• Hindi – Singapore
• Bengali – Singapore