PLS Predict: Questions about Information Theoretic Criteria
Posted: Fri Jun 29, 2018 8:59 pm
Greetings,
I am writing a paper using the PLS Predict function, as well as the information theoretic criteria advocated by Sharma et al. (forth coming) (see the following link for the Excel sheet to calculate these indices: https://www.pls-sem.net/downloads/). I am comparing multiple models in the literature I am researching, and using RMSE and MAE (for predictive ability) and BIC and GM (for the balance between explanation and prediction). Here are a few questions.
(a) What exactly do we mean by "saturated model R2" in the Excel sheet for the information theoretic criteria calculation, WHEN you have 2nd-order measurement models? Should lower-order variables also directly predict the target outcome variable, or only the higher-orders (I assumed the latter is the case, but am asking because of the following problems)?
(b) I have negative BIC (as well as many other indices in negative, such as AIC, AICu, HQ, HQc). What does this mean? Indeed, the Excel file has a negative value for BIC when you download it. Is there any problems with the equations, or is this just fine? If it's fine, how to we interpret negative values in model comparisons (e.g., closer to zero is better, or negative values meaning those models are crappy anyway?).
(c) I have one model that has a moderated path to the target outcome variable. And the RMSE and MAE for this model is remarkably lower (and thus better prediction) than other models'. However, this moderated model has lower explanation power. I understand that explanation and prediction are two different things. Would this be one of those cases where low explanatory power models predict well, OR is there any problems around applying PLS Predict and RMSE/MAE to moderated models?
I would appreciate any comments!
All the best,
Shin
I am writing a paper using the PLS Predict function, as well as the information theoretic criteria advocated by Sharma et al. (forth coming) (see the following link for the Excel sheet to calculate these indices: https://www.pls-sem.net/downloads/). I am comparing multiple models in the literature I am researching, and using RMSE and MAE (for predictive ability) and BIC and GM (for the balance between explanation and prediction). Here are a few questions.
(a) What exactly do we mean by "saturated model R2" in the Excel sheet for the information theoretic criteria calculation, WHEN you have 2nd-order measurement models? Should lower-order variables also directly predict the target outcome variable, or only the higher-orders (I assumed the latter is the case, but am asking because of the following problems)?
(b) I have negative BIC (as well as many other indices in negative, such as AIC, AICu, HQ, HQc). What does this mean? Indeed, the Excel file has a negative value for BIC when you download it. Is there any problems with the equations, or is this just fine? If it's fine, how to we interpret negative values in model comparisons (e.g., closer to zero is better, or negative values meaning those models are crappy anyway?).
(c) I have one model that has a moderated path to the target outcome variable. And the RMSE and MAE for this model is remarkably lower (and thus better prediction) than other models'. However, this moderated model has lower explanation power. I understand that explanation and prediction are two different things. Would this be one of those cases where low explanatory power models predict well, OR is there any problems around applying PLS Predict and RMSE/MAE to moderated models?
I would appreciate any comments!
All the best,
Shin