Page 1 of 1

PLS Predict: interpreting RSME, MAE, and MAPE

Posted: Sat Dec 10, 2016 7:16 am
by jjdekker
As the resources page on SmartPLS.de does not yet provide any guidance on how to interpret RSME, MAE, and MAPE.What are acceptable scores for these three measures and why?

Thanks in advance.
Hans

Re: PLS Predict: interpreting RSME, MAE, and MAPE

Posted: Sun Dec 11, 2016 8:33 pm
by jmbecker
There are no acceptable scores or thresholds for these values. These criteria are used to compare models. The smaller the values the better the model.

Re: PLS Predict: interpreting RSME, MAE, and MAPE

Posted: Tue Dec 20, 2016 5:29 pm
by Piddzilla
jmbecker wrote:There are no acceptable scores or thresholds for these values. These criteria are used to compare models. The smaller the values the better the model.
Isn't the mean absolute percentage error (MAPE) an indication of the uncertainty of the prediction?

If the MAPE of a model displays a value of, say, 2.5 %, then we can say that the evaluated data varies 2.5 % from our predicted model. In other words, our prediction was 97.5 % correct. If that is good or bad depends on the specific situation - sometimes being 60 % correct is good enough and sometimes being 99 % correct is not good enough. However, when comparing models that are supposed to predict the same thing, 2 % is of course "better" than 2.5 %. But is it significantly "better"?

When it comes to RSME and MAE I have not quite yet figured out how to interpret them besides the fact, as jmbecker stated, "the lower, the better". But I think that RSME should be related to the units used to measure data. That is, an RSME value corresponding approximately to (MAPE = 2 %) could be a value of 0.01 just as well as a value of 1 000, all depending on the size (and the mean) of the units. The MAPE produces percantages while the MAE and the RSME produces absolute values (I think).

Again, when comparing the predictive quality of different models, the model displaying the lowest RMSE has the higher predictive quality.

Since RSME, MAE, and MAPE, respectively, has their weaknesses (one is too conservative and another is too liberal, sort of), they should be considered together when assessing predictive qualities of a model.

These are all speculations - I initially went here myself to look for answers to these questions. :)

EDIT: Just ran the PLS predict algorithm on the PLS-SEM BOOK: Corporate Reputation Extended model to see what happened. The RSME and the MAE measures seem to measure the "same thing", although the RSME values are always higher (more conservative) compared to the MAE measures (more liberal). Among the constructs in the model, CUSA displays the lowest values for all three criteria (RSME = 1.1030, MAE = 0.799, and MAPE = 18.6 %).