Dria
Swan
NFTHardhat
21,000 USDC
View results
Previous Next
Submission Details
Severity: high
Invalid

`validationDeviationFactor` is not used in Outliers calculations

alqaqa

Description

When checking for outliers, we check that there values lies in the range if mean -/+ standard deviation.

In Statistics, This equation (mean -/+ standard deviation) is incomplete, as not all data behavs the same, and this boundry for outliers can be infair. The complete Formula is: mean -/+ factory * standard_deviation, where the Factory is used to extend or norrow the correct region.

In our Code, we are initializing validationDeviationFactor by 2, and generationDeviationFactor by 1.

llm/LLMOracleManager.sol#L54

function __LLMOracleManager_init(uint256 _platformFee, uint256 _generationFee, uint256 _validationFee) ... {
...
​
>> validationDeviationFactor = 2;
generationDeviationFactor = 1;
​
...
}

The problem is that we are not using validationDeviationFactor when making the Outliers check (lower and upper) for validators, which will make it behave as if the factory is 1, which is totally incorrect as it should be two. But the Check is implemented Correctly when checking generators outliers.

llm/LLMOracleCoordinator.sol#L343 | llm/LLMOracleCoordinator.sol#L368

function finalizeValidation(uint256 taskId) private {
...
for (uint256 g_i = 0; g_i < task.parameters.numGenerations; g_i++) {
...
for (uint256 v_i = 0; v_i < task.parameters.numValidations; ++v_i) {
uint256 score = scores[v_i];
// @audit `validationDeviationFactor` is missing
>> if ((score >= _mean - _stddev) && (score <= _mean + _stddev)) { ... }
}
...
}
...
for (uint256 g_i = 0; g_i < task.parameters.numGenerations; g_i++) {
// ignore lower outliers
// @audit `generationDeviationFactor` is presented
>> if (generationScores[g_i] >= mean - generationDeviationFactor * stddev) {
_increaseAllowance(responses[taskId][g_i].responder, task.generatorFee);
}
}
}

This will make Outliers detection goes incorrectly, by narrowing the range of errors for validations, making the process unfair for Validators.

So the non-outliers will be 68% only instead of 95% of validators who joined the validation for this task according to the empirical rule for normal distribution.

Recommendations

Add the validationDeviationFactor to the calculations.

diff --git a/contracts/llm/LLMOracleCoordinator.sol b/contracts/llm/LLMOracleCoordinator.sol
index 1ba2176..68e1427 100644
--- a/contracts/llm/LLMOracleCoordinator.sol
+++ b/contracts/llm/LLMOracleCoordinator.sol
@@ -340,7 +340,10 @@ contract LLMOracleCoordinator is LLMOracleTask, LLMOracleManager, UUPSUpgradeabl
uint256 innerCount = 0;
for (uint256 v_i = 0; v_i < task.parameters.numValidations; ++v_i) {
uint256 score = scores[v_i];
- if ((score >= _mean - _stddev) && (score <= _mean + _stddev)) {
+ if (
+ (score >= _mean - validationDeviationFactor * _stddev) &&
+ (score <= _mean + validationDeviationFactor * _stddev))
+ {
innerSum += score;
innerCount++;
Updates

Lead Judging Commences

inallhonesty Lead Judge 11 months ago
Submission Judgement Published
Invalidated
Reason: Non-acceptable severity

Support

FAQs

Can't find an answer? Chat with us on Discord, Twitter or Linkedin.