Good AI needs
thoughtful people.

What you’ll do

• Create realistic evaluation questions, reference answers, and supporting source passages from authorized documentation.
• Review model responses for correctness, evidence support, citation validity, completeness, and appropriate handling of uncertainty.
• Distinguish retrieval gaps from answer-generation errors when the available traces support that distinction.
• Apply written scoring guidelines consistently, document your rationale, and flag ambiguous or conflicting evidence.
• Review cases independently, discuss disagreements, and record how final judgments were reached.
• Maintain organized evaluation datasets and contribute reproducible examples to failure reports.

Minimum requirements

• At least 1 year of relevant experience in AI evaluation, data annotation, QA, technical research, or an equivalent portfolio of practical work.
• A working understanding of LLMs, retrieval-augmented generation, hallucinations, and source citations.
• Strong English reading and writing skills; able to explain a judgment clearly and concisely.
• Careful document analysis and the ability to follow a rubric without filling gaps with assumptions.
• Comfort working with spreadsheets and structured data such as CSV or JSONL.
• Reliable remote communication and responsible handling of confidential project information.

Nice to have

• Python or SQL for dataset checks, sampling, or lightweight analysis.
• Experience with evaluation tools, reviewer agreement analysis, or quality assurance workflows.
• Domain expertise in technical documentation, customer support, or enterprise knowledge bases.
• Bangla–English language skills or experience evaluating multilingual answers.

These are optional strengths, not required qualifications.

What you’ll do

• Design and implement backend services, APIs, data pipelines, and internal tools for evaluation delivery.
• Build versioned ingestion and evaluation workflows with clear validation, failure handling, and audit trails.
• Integrate model APIs and retrieval systems while accounting for latency, cost, rate limits, and changing outputs.
• Write maintainable code, meaningful automated tests, and concise technical documentation.
• Investigate production issues and improve observability, performance, security, and operational reliability.
• Review code, mentor teammates, and collaborate with evaluators to translate practical needs into software.

Minimum requirements

• At least 5 years of professional software engineering experience, or equivalent demonstrated ownership of substantial production systems.
• Strong Python skills and experience building backend services with FastAPI, Django, or a comparable framework.
• Solid SQL and relational database skills, including schema design, migrations, and query performance.
• Experience designing APIs, testing services, reviewing code, and working with Git and CI/CD.
• Practical experience with containers and deploying or operating software in a cloud environment.
• Clear written communication, independent problem solving, and accountability for delivery in a remote team.

Nice to have

• Experience building RAG or LLM applications, search systems, or evaluation infrastructure.
• PostgreSQL, vector search, asynchronous jobs, queues, and data processing at scale.
• TypeScript, React, or Next.js for internal review interfaces.
• Experience with access controls, sensitive data workflows, observability, or infrastructure as code.

These are optional strengths, not required qualifications.

What you’ll do

• Translate business and evaluation needs into an architecture, technical roadmap, and achievable delivery milestones.
• Make and document engineering tradeoffs across quality, cost, performance, security, and delivery time.
• Lead design and code reviews, establish engineering standards, and contribute hands-on implementation where it matters.
• Mentor engineers, clarify ownership, remove blockers, and coordinate work across engineering and evaluation.
• Own technical risk assessment, release readiness, incident follow-up, and improvements to system reliability.
• Explain technical decisions and delivery constraints clearly to the team, collaborators, and clients.

Minimum requirements

• At least 7 years of software engineering experience, including 2 years leading technical delivery or equivalent demonstrated leadership.
• A record of designing, delivering, and operating production backend or data-intensive systems.
• Strong hands-on backend engineering skills, preferably in Python, with sound API and database design judgment.
• Experience guiding engineers through architecture decisions, code review, planning, and technical disagreements.
• Working knowledge of cloud infrastructure, CI/CD, observability, security practices, and incident management.
• Strong written communication and the ability to lead a distributed team through unclear requirements.

Nice to have

• Technical leadership on RAG, LLM, search, AI evaluation, or human review platforms.
• Experience building a small team or establishing engineering practices in an early-stage company.
• Experience scoping client engagements and communicating delivery risks to non-technical stakeholders.
• Experience with evaluation methodology, dataset quality, model cost controls, or privacy-conscious architecture.

These are optional strengths, not required qualifications.