Submission Details
Severity:
`LLMOracleCoordinator.getBestResponse()` Function Doesn't Validate The Returned Result Against A Minimum Score
## Summary
`LLMOracleCoordinator.getBestResponse()` function is supposed to return the generator output that got the highest validation score amongst the other generators for the same task, but this wouldn't prevent consuming a result with a score of zero as there's no check for a minimum acceptable score for the generators (when the validation is required for that task, i.e `task.parameters.numValidations > 0`).
## Vulnerability Details
- `LLMOracleCoordinator.getBestResponse()` function is called when a buyer agent calls `BuyerAgent.purchase()` to move on with the purchase process of the requested assets that has been listed to him, and this is done after the agent calls `BuyerAgent.oraclePurchaseRequest()` with the required input (listed assets to be purchased), where a call is made to the `LLMOracleCoordinator.request()` to make a purchase request after paying the process fees.
- The oracle request is first processed by the generators adding their responses (via `LLMOracleCoordinator.respond()`) --> then after the responses generation is done, the task is moved to the validation phase to be validated and finalized after getting the required validations number -if the task requires a validation- (via `LLMOracleCoordinator.validate()`).
- In the validation phase: each validator gives a score for all generators responses, and after that the validation is finalized via `LLMOracleCoordinator.finalizeValidation()`.
- In `LLMOracleCoordinator.finalizeValidation()`; the validators scores for each generator response is processed and the scores for each generator is accepted if it's within the acceptable calculated range (mean+- standard deviation) , then the scores of all generators are processed and generators with scores within the calculated range will be rewarded.
- But it was noticed that during the validation/finalizeValidation process, there's no check if all the generators got a zero score as the following check will be satisfied ([L343](https://github.com/Cyfrin/2024-10-swan-dria/blob/c8686b199daadcef3161980022e12b66a5304f8e/contracts/llm/LLMOracleCoordinator.sol#L343)) `if ((score >= _mean - _stddev) && (score <= _mean + _stddev))` when the mean and standard deviation are zeros:
```javascript
function finalizeValidation(uint256 taskId) private {
//...
// compute score for each generation
for (uint256 g_i = 0; g_i < task.parameters.numGenerations; g_i++) {
// get the scores for this generation, i.e. the g_i-th element of each validation
uint256[] memory scores = new uint256[]();
for (uint256 v_i = 0; v_i < task.parameters.numValidations; v_i++) {
scores[v_i] = validations[taskId][v_i].scores[g_i];
}
// compute the mean and standard deviation
(uint256 _stddev, uint256 _mean) = Statistics.stddev(scores);
uint256 innerSum = 0;
uint256 innerCount = 0;
for (uint256 v_i = 0; v_i < task.parameters.numValidations; ++v_i) {
uint256 score = scores[v_i];
@L343 if ((score >= _mean - _stddev) && (score <= _mean + _stddev)) {
innerSum += score;
innerCount++;
// send validation fee to the validator
_increaseAllowance(validations[taskId][v_i].validator, task.validatorFee);
}
}
// set score for this generation as the average of inner scores
uint256 inner_score = innerCount == 0 ? 0 : innerSum / innerCount;
responses[taskId][g_i].score = inner_score;
}
//...
}
```
## Impact
So `LLMOracleCoordinator.getBestResponse()` would result in getting a result that hasn't got enough validation score when the mean and standard deviation are zeros (for a large number of generators with low scores or when all generators got zero scores).
## Proof of Concept
[LLMOracleCoordinator.getBestResponse() function](https://github.com/Cyfrin/2024-10-swan-dria/blob/c8686b199daadcef3161980022e12b66a5304f8e/contracts/llm/LLMOracleCoordinator.sol#L404C5-L423C6)
```javascript
function getBestResponse(uint256 taskId) external view returns (TaskResponse memory) {
TaskResponse[] storage taskResponses = responses[taskId];
// ensure that task is completed
if (requests[taskId].status != LLMOracleTask.TaskStatus.Completed) {
revert InvalidTaskStatus(taskId, requests[taskId].status, LLMOracleTask.TaskStatus.Completed);
}
// pick the result with the highest validation score
TaskResponse storage result = taskResponses[0];
uint256 highestScore = result.score;
for (uint256 i = 1; i < taskResponses.length; i++) {
if (taskResponses[i].score > highestScore) {
highestScore = taskResponses[i].score;
result = taskResponses[i];
}
}
return result;
}
```
## Tools Used
Manual Review.
## Recommendations
Update `LLMOracleCoordinator.getBestResponse()` to check against a minimum score of the returned result (in case validation is required for the task).
Updates
Prize pool breakdown
Total prize
21,000 USDC
nSLOC
1,03420 USDC / LOC
20,000 USDC
1,000 USDC
Live
The contest is live. Earn rewards by submitting a finding.
Appeals
This is your time to appeal against judgements on your submissions.
Appeals Review
Appeals are being carefully reviewed by our judges.