Scoring Algorithm Overview

Your score is calculated in three levels:

1. Criteria Level: Each criterion in a parameter can be answered as "Yes", "No", or "NA" (not applicable). Every criterion has point values for both Yes and No answers. When evaluated, you earn the corresponding points based on the answer, while NA responses are excluded from scoring.

2. Parameter Level: Each parameter's score is calculated as a percentage: the points earned from its criteria divided by the maximum possible points. For example, if a parameter has 3 criteria worth a maximum of 10 points total, and you earned 7 points, your parameter score would be 70%.

3. Playbook Level: The overall playbook score is a weighted average of all parameter scores. Parameters with more criteria (and thus more total possible points) carry more weight in the final score. This ensures that more comprehensive parameters have greater impact on your overall performance.

Note: If any criterion marked as "fatal" receives a failing answer, the entire playbook automatically scores 0%, regardless of other criteria performance.

Example 1

Parameter A has 2 criteria:

• Criterion 1: Greeted customer properly? (Yes=5pts, No=0pts) → Answer: Yes → 5 points

• Criterion 2: Used customer's name? (Yes=5pts, No=0pts) → Answer: No → 0 points

• Parameter A Score: 5/10 = 50%

Parameter B has 1 criterion:

• Criterion 3: Resolved the issue? (Yes=10pts, No=0pts) → Answer: Yes → 10 points

• Parameter B Score: 10/10 = 100%

Parameter C has 1 criterion:

• Criterion 4: Used inappropriate language? (Yes=0pts, No=10pts) → Answer: No → 10 points

• Parameter C Score: 10/10 = 100%

Playbook Score: (50% × 10 points + 100% × 10 points + 100% × 10 points) / 30 total points = 83.33%

This example shows how criteria can be structured to penalize unwanted behaviors - Parameter C asks about inappropriate language where avoiding it (answering "No") earns the full points, while doing it (answering "Yes") earns zero points.

Example 2

Parameter A has 2 criteria:

• Criterion 1: Greeted customer properly? (Yes=5pts, No=0pts) → Answer: Yes → 5 points

• Criterion 2: Used customer's name? (Yes=5pts, No=0pts) → Answer: NA (customer didn't provide name)

• Parameter A Score: 5/5 = 100% (only Criterion 1 counted)

Parameter B has 1 criterion:

• Criterion 3: Resolved the issue? (Yes=10pts, No=0pts) → Answer: Yes → 10 points

• Parameter B Score: 10/10 = 100%

Playbook Score: (100% × 5 points + 100% × 10 points) / 15 total points = 100%

This example shows how NA responses are completely excluded from scoring - they don't count toward earned points OR maximum possible points. This prevents penalizing agents for situations that don't apply to their specific conversation.

Example 3

Parameter A has 2 criteria:

• Criterion 1: Greeted customer properly? (Yes=5pts, No=0pts) → Answer: Yes → 5 points

  
  

• Criterion 2: Used customer's name? (Yes=5pts, No=0pts) → Answer: No → 0 points

  
  

• Parameter A Score: 5/10 = 50%

Parameter B has 1 criterion:

• Criterion 3: [FATAL] Disclosed customer's private information? (Yes=0pts, No=10pts) → Answer: Yes → FATAL FAILURE

Playbook Score: 0% (entire conversation fails due to fatal criterion)

This example shows how fatal criteria represent critical requirements or policy violations. If any fatal criterion fails, the entire playbook receives a score of 0%, regardless of performance on all other criteria. This ensures that certain unacceptable behaviors always result in a failing evaluation.