Conversation and evaluation design

SCL uses two related but distinct designs:

• The simulated-customer design controls how the SCAI character behaves during the conversation.
• The evaluation criteria control how the completed transcript and elapsed time are assessed.

They should be reviewed together but versioned independently when different systems or owners manage them.

Design the simulated customer

The design-team handoff should document at least:

Company or business context:
Scenario description and learning objective:
Opening message:
AI role and prohibited roles:
Customer goals and hidden constraints:
Facts the customer knows, may reveal, and must not invent:
Typical questions and objections:
How behaviour changes in response to the learner:
Closing behaviour:
Guardrails:
Optional start action and trigger:
Structured persona/user data:
Persona name, communication style, personality, and background:
Required languages and voice needs:

State repeatedly where ambiguity is possible that the AI acts as the customer and the human acts as the employee. Test that the character gives information after effective questions, remains appropriately vague otherwise, raises one objection at a time, and creates opportunities for every important learner behaviour. An indicator cannot be assessed fairly if the conversation never gives the learner a realistic chance to demonstrate it.

Understand the evaluation target

The runtime contract uses counterintuitive speaker names:

Criteria and test transcripts must preserve this mapping. Reversing it produces plausible-looking but invalid feedback.

Define performance indicators

Each indicator should specify:

• A unique, learner-friendly name.
• The observable behaviour being assessed.
• Positive and negative evidence from the transcript.
• A 1–5 scoring rubric with clear boundaries.
• Its weight in the overall result.
• Any conditions under which it is not assessable.

The evaluation prompt instructs the model to return indicators in the same order as the criteria and classify scores 4–5 as achieved and 1–3 as improvable. The backend currently checks only that the indicator list is non-empty; test these additional rules explicitly.

Use deterministic rules where appropriate

Measures such as elapsed time should use explicit formulas and non-overlapping ranges. For example:

target = 240 seconds
deviation = absolute value of elapsed time - target

0-15 seconds from target: 5
16-30 seconds from target: 4
31-60 seconds from target: 3
61-90 seconds from target: 2
more than 90 seconds from target: 1

Define how decimal weighted results are rounded. The prompt requests integer overall and indicator scores, while the backend directly enforces the overall 1–5 integer range.

Current feedback rules

The backend validates an overall score from 1 to 5, a non-empty description, and at least one performance indicator. Its active core instructions also require:

• Evidence only from the supplied transcript and criteria.
• All output text in the requested evaluation language.
• Specific strengths and improvements based on learner messages.
• At least one concrete action-plan item.
• No numeric score repeated inside descriptive text.

The API contract can carry summary items and recommended courses, but the current core output schema does not request them. Do not promise those sections for a role-play until its evaluation path has been tested to return them.

Evaluation criteria template

Evaluation context
- Domain, scenario, target learner, and assessment purpose
- API speaker Customer = human learner being evaluated
- API speaker Agent = AI simulated customer used only as context

Phase or indicator 1
- Observable behaviour:
- Evidence to reward:
- Missing or poor evidence:
- Score 1:
- Score 2:
- Score 3:
- Score 4:
- Score 5:
- Weight:

Repeat for each indicator.

Elapsed-time indicator, if used
- Target in seconds:
- Absolute-deviation formula:
- Non-overlapping score bands from 1 to 5:
- Required description of elapsed time and applied rule:

Overall score calculation and rounding:
Weights, totaling exactly 100%:
Required strengths and improvements:
Action-plan expectations:
Language and terminology requirements:
Rules for short, partial, or low-quality transcripts:
Safety, fairness, and bias checks:

Keep phase names, order, and weights stable within a released version. Verify the total weight mechanically and reconcile any elapsed-time target with the published SessionInfo.

Test before activation

Use a balanced set of transcripts:

• Strong, average, and weak performance.
• Missing phases and early termination.
• Silence, recognition errors, and very short answers.
• Learner and simulated-customer messages that could be confused.
• Exact timing boundaries for any formula-based indicator.
• Every supported feedback language.

Review not only the overall score but also evidence attribution, indicator order, status, language consistency, and actionability. Keep the expected outputs with the released criteria as regression evidence.