Dria
Swan
NFTHardhat
21,000 USDC
View results
Previous Next
Submission Details
Severity: medium
Valid

Validation Scoring System Can Be Gamed by Malicious Validators

0xnbvc

Vulnerability Details

The validation scoring system in LLMOracleCoordinator can be gamed by malicious validators who wait to submit their scores last. The issue occurs because:

  1. All validations are publicly visible on-chain

  2. There's no time limit for submitting validations

  3. The scoring mechanism is deterministic and uses mean/standard deviation

  4. Validators get paid if their scores fall within 1 standard deviation of the mean

Here's how a malicious validator could exploit this:

  1. Wait for other validators to submit their scores first

  2. Calculate the current mean and standard deviation from the public validation data

  3. Submit scores that are guaranteed to fall within the acceptable range

  4. Get paid the validatorFee despite not providing honest validation

LLMOracleCoordinator.sol#L334-L355

// compute the mean and standard deviation
(uint256 _stddev, uint256 _mean) = Statistics.stddev(scores);
​
// compute the score for this generation as the "inner-mean"
// and send rewards to validators that are within the range
uint256 innerSum = 0;
uint256 innerCount = 0;
for (uint256 v_i = 0; v_i < task.parameters.numValidations; ++v_i) {
uint256 score = scores[v_i];
if ((score >= _mean - _stddev) && (score <= _mean + _stddev)) {
innerSum += score;
innerCount++;
​
// send validation fee to the validator
_increaseAllowance(validations[taskId][v_i].validator, task.validatorFee);
}
}
​
// set score for this generation as the average of inner scores
uint256 inner_score = innerCount == 0 ? 0 : innerSum / innerCount;
responses[taskId][g_i].score = inner_score;
}

Impact

  • Malicious validators can get paid without providing honest validations

  • The quality of the validation system is compromised

  • Honest validators are disadvantaged compared to those who game the system

  • The protocol's ability to select the best LLM responses is undermined

Recommendations

Implement a commit-reveal scheme for validations:

  • Validators first submit a hash of their scores

  • Only after all validators have committed, they reveal their actual scores

  • This prevents later validators from knowing earlier scores

Updates

Lead Judging Commences

inallhonesty Lead Judge 10 months ago
Submission Judgement Published
Validated
Assigned finding tags:

Unbounded score values in `validate` function

Support

FAQs

Can't find an answer? Chat with us on Discord, Twitter or Linkedin.