Skip to content
Book a demo

Hire an AI Evaluation Specialist.

AI tested on real cases before it reaches customers.

Also called: LLM Evaluation Specialist, AI QA Tester, AI Quality Analyst

From AUD $8 an hourAUD $8-35/hr

Exact rate on your call. All-in.

  • SaaS & tech
  • E-commerce
  • Financial services
Book a demoTalk now on Teams

A free 15-minute call on Teams.

  • Works your hours
  • Philippines hub, 2 to 3 hours behind Sydney
  • Month to month
What your AI Evaluation Specialist gets doneSample day
  • Builds test sets of 50-500 cases from real, anonymised ticketsTest set design
  • Runs evaluations in Promptfoo, LangSmith, LangfuseRunning evaluations
  • Try to make assistants leak instructions or personal dataRed-teaming & safety
  • Writes a ship or hold recommendation for each release with the numbersReporting & release checksDoing
In your tools
  • Microsoft Foundry evaluations
  • Python
  • Jupyter
  • Google Sheets
  • Microsoft Excel
  • Looker Studio

Pick the level. See the range.

AUD $8-13an hourExact rate on your call

Level 2 of 4, also called mid-level.Recommended

Builds and maintains test sets of 100-500 cases, writes rubrics and runs evaluations.

  • Gold test set with rubric and expected answers
  • Release report comparing two prompt or model versions
  • LLM-as-judge calibration report

1-3 years in QA, annotation, analytics or AI operations.

How the Intermediate level works

Per hour, all-in. Exact rate confirmed on your call. Prices exclude GST.

What you hand to your AI Evaluation Specialist

Tasks

  • Test set design

    • Builds test sets of 50-500 cases from real, anonymised tickets
    • Writes marking rubrics with clear pass and fail examples
    • Keeps a gold set marked by the client's subject expert

  • Running evaluations

    • Runs evaluations in Promptfoo, LangSmith, Langfuse
    • Sets up LLM-as-judge checks
    • Marks outputs by hand where automated checks are not reliable
    • Samples live conversations or agent runs each week

  • Red-teaming & safety

    • Try to make assistants leak instructions or personal data
    • Tests prompt injection through documents, emails and web pages for agents
    • Logs every finding with severity, steps to reproduce and a suggested fix

  • Reporting & release checks

    • Writes a ship or hold recommendation for each release with the numbers
    • Reports the top failure categories monthly with an owner
    • Compares models on quality, cost and response time

Tested before you meet them

How we test AI work

Tested on AI work, with mistakes planted in it

  • A real task from the roleDone with the AI tools the job uses, in a suite like yours.
  • Mistakes planted in the AI's outputCatching them is part of the pass mark, at every level.
  • Restricted data stays outPutting it into an AI tool fails the test, whatever the score.
Sample result, not a real candidate
AI workflow test: Failure-handling review60 minutes · marked by a senior
Planted problems found
2 of 2
Error handling and a retry added
Yes
Personal data out of the logs
Yes
Alert on bad input
Yes

Passed at IntermediateReal profiles show the date and score.

  1. Rubric marking testAll levels45 min
  2. Test set designIntermediate and up1.5 hrs
  3. Evaluation tool testIntermediate and up60 min
  4. Red-team exerciseIntermediate and up60 min
  5. Results write-upAll levels40 min
  6. Evaluation strategy role-playExpert and Specialist60 min
  7. AI workflow test: Failure-handling review60 min

One score on every profileSample shown

89%

Three checks before your shortlist

  • Skills tested: A skills test for the role, marked by a senior
  • English and comms: written English tested, spoken English checked on a call
  • ID, work history and references: photo ID, past roles and two referees

From booked call to day one

  1. Step 1: Book a demo

    Tell us the role, your hours and your tools.

  2. Step 2: Meet your shortlist

    Their level and hourly price, tested before you meet.

  3. Step 3: Pick your person

    Month to month. We handle the contract, the equipment and the onboarding, and make sure it all runs.

  4. Step 4: They start work

    In your tools, within 12 to 48 hours of a successful interview.

Questions answered

It depends on the level and the work. Intermediate sits in the AUD $8-13 an hour range; Expert sits in the AUD $13-20 an hour range; Specialist sits in the AUD $20-35 an hour range. We confirm your exact rate on the call, all-in and month to month, before you meet anyone.

Test set design, running evaluations, red-teaming & safety and reporting & release checks, run to your priorities. You choose what they own first.

Seven practical tests, marked by a senior, including rubric marking test, test set design and evaluation tool test. You see the score before you meet.

Tell us and we replace them free, with a new shortlist of tested people to meet. Seats are month to month with 30 days' notice.

More questions

Within 12 to 48 hours of a successful interview. First you meet your shortlist, interview the people you like and pick your person.

Yes. Your AI Evaluation Specialist works your business hours in your time zone and joins your Teams or Slack like anyone else. The hub matched to Australia & NZ clients is in the Philippines, 2 to 3 hours behind Sydney.

You do. Your AI Evaluation Specialist builds in your own accounts (Zapier, Make or n8n, Microsoft 365 or Google Workspace, your AI provider), under your logins and billing. Each build is logged with notes and a recorded walkthrough, so it stays with you if you leave us.

You do, on your own accounts, at the provider's price. We never mark it up. Your AI Evaluation Specialist asks before adding a paid tool and sets spending limits where the tool allows.

Every automation has error alerts, and your AI Evaluation Specialist fixes or escalates failures within the day. Steps that send, pay or change records wait for a person's approval, so a failure stops the run instead of sending the wrong thing.

You agree what gets built first. Builds are tested on real cases, and nothing goes live until you or your nominee says yes. Every change is logged: what, when and why.

Yes. Your AI Evaluation Specialist works inside your Microsoft 365 or Google Workspace with the access your admin gives, so Copilot and Gemini follow your tenant's rules and AI policy. You can remove access at any time.

An Intermediate AI Evaluation Specialist builds and maintains test sets of 100-500 cases, writes rubrics and runs evaluations. An Expert AI Evaluation Specialist owns evaluation for a product or a set of agents: designs release checks and automates evaluations. We usually recommend Intermediate.

It stays under your rules: your admin settings and AI policy apply. Our talent rules ban client data in personal or free AI accounts, and passwords, card numbers, health details and ID numbers never go into any AI tool.

You direct the work day to day. We handle recruiting, equipment and quality checks.

Yes. Every AI Evaluation Specialist is tested live in the tools on your job description, such as Braintrust and Arize Phoenix, before you meet them.

In week one: client's approved AI tools and plans confirmed in writing; read-only access to AI logs, traces and conversation history; and subject-matter expert named on the client side. You agree the first priorities with them on day one.

Yes. It's also called LLM Evaluation Specialist, AI QA Tester and AI Quality Analyst. Same tasks, tests and prices, whatever the name.

Their written English, with a scored test, and their spoken English, on a call with us. We also check their ID, work history and references.

Yes. You meet your shortlist on a short call and pick the person you want. Nobody joins without your yes.

In our talent hubs in the Philippines, Vietnam, Kenya, Nigeria, Brazil and Colombia, and we hire from 60+ countries. Wherever they are, they work your hours.

Pay only for the hours they work. All your Seats are on one bill, and Studio plans are billed monthly.

It can, at most once a year. Your person's hourly rate may rise by up to 5% a year, from the first anniversary of the Seat's start, with 60 days' notice and the right to end the Seat before it applies.

Only when you and your person both agree to it in advance. Otherwise they work the hours you set, and you pay for the hours they log.

You aren't charged for hours your person doesn't work, apart from paid leave. After their third month, your person earns 8 hours of paid leave a month. Paid leave hours are charged at their usual rate.

Your person logs their hours each day in the Get Hands client app, where you can see them. Your next invoice is adjusted to those hours.

Yes, with no minimum hours. From 20 to 37 hours a week, the hourly rate is 10% higher than full-time. Under 20 hours, it's 20% higher.

Yes. Tell us the roles on a call and we shortlist tested people for each.

A tested shortlist within 72 hours of your call, or your first week is free. That's part of The Right Hands Guarantee.

Yes. Call us on +61 3 4421 6491, or choose Talk now on the booking form for a Microsoft Teams link straight away, any time.

Book a 15-minute demo on Microsoft Teams, any time, day or night. We show you the people, levels and prices that fit. No obligation.

Meet the people who fit your work.

Tell us the role on a 15‑minute Teams call. We'll show you who fits, or tell you straight if we can't fill it.

Book a demoTalk now on Teams

Talk now is live on Teams, any time.