q² effect size

 PLS Junior User
 Posts: 2
 Joined: Tue Jan 16, 2018 9:39 am
 Real name and title: Mr. Imdadulah
Re: Some methodological issues
When assessing q², I get negative values. Can I ignore the negative symbol and interpret the magnitude of the value whether positive or negative?

 SmartPLS Developer
 Posts: 820
 Joined: Tue Mar 28, 2006 11:09 am
 Real name and title: Dr. JanMichael Becker
Re: Some methodological issues
Generally, q² can also be negative unlike f².
The R² (i.e., explained variance) always improves if you add more predictors (and therefore f² is always positive).
The predictive relevance (Q²) of a model can also decrease if you add predictors, due to overfitting of the model (e.g., adding irrelevant noisy predictors).
Thus, theoretically, a negative q² means that removing the focal variable actually improves the predictive power of the model (Q²). That is, in terms of predictions, you are better off without the focal variable.
However, there are some potential concerns with calculating q² in a PLS model (which are also the reason why we have not implemented such a calculation).
For example, by deleting a latent variable you are changing the model and thus the PLS estimates might change as they are dependent on the structural model. Whether this is good or not (correct or not) when calculating q² has never been finally decided.
Deleting variables from the PLS model may change the order of manifest variables in the missing value substitution and prediction process. This may positively or negatively effect the predictive relevance without having a reason in the deletion of the focal variable.
Therefore, the change in Q² may not be attributable only to the inclusion/exclusion of the focal variable. Thus, q² might not be a reliable measure, which is why I usually do not recommend it.
At the moment, more research is necessary to more fully understand this type of analysis and how it can be correctly applied.
The R² (i.e., explained variance) always improves if you add more predictors (and therefore f² is always positive).
The predictive relevance (Q²) of a model can also decrease if you add predictors, due to overfitting of the model (e.g., adding irrelevant noisy predictors).
Thus, theoretically, a negative q² means that removing the focal variable actually improves the predictive power of the model (Q²). That is, in terms of predictions, you are better off without the focal variable.
However, there are some potential concerns with calculating q² in a PLS model (which are also the reason why we have not implemented such a calculation).
For example, by deleting a latent variable you are changing the model and thus the PLS estimates might change as they are dependent on the structural model. Whether this is good or not (correct or not) when calculating q² has never been finally decided.
Deleting variables from the PLS model may change the order of manifest variables in the missing value substitution and prediction process. This may positively or negatively effect the predictive relevance without having a reason in the deletion of the focal variable.
Therefore, the change in Q² may not be attributable only to the inclusion/exclusion of the focal variable. Thus, q² might not be a reliable measure, which is why I usually do not recommend it.
At the moment, more research is necessary to more fully understand this type of analysis and how it can be correctly applied.
Dr. JanMichael Becker, University of Cologne, SmartPLS Developer
Researchgate: https://www.researchgate.net/profile/Ja ... v=hdr_xprf
GoogleScholar: http://scholar.google.de/citations?user ... AAAJ&hl=de
Researchgate: https://www.researchgate.net/profile/Ja ... v=hdr_xprf
GoogleScholar: http://scholar.google.de/citations?user ... AAAJ&hl=de

 PLS Junior User
 Posts: 1
 Joined: Mon Jul 09, 2018 1:50 am
 Real name and title: MichaelMug
q effect size
Hey Dev, thanks for getting back to me. This is actually a zstatistic, along with a pvalue. I'm not sure what the relationship is to a standardized zscore?? Maybe I didn't state the question correctly So does that change anything regarding making it possible to convert to an effect size? I know if you have an Fstatistic with a pvalue, there is a method to convert it with your software, for example to an effect size without having the means and SD's  was wondering if there is anything similar for a zstatistic?? I've been searching online and reading all my old stats books but can't come up with anything.
2nd question: I do have the mean and standard deviation of the immediate and delayed recall, is that what you are asking? However, I am not looking at the difference between these two variables, I'm using them to calculate percent of material recalled at delay  mean score on delayed recall divided by mean score on immediate recall x 100. Previously, I was looking at the difference between these two variables, but I ran in too many floor effects so I changed the variable to "percent retention." So, there is not a way to estimate SD in this case other than using what is found in the literature, right?
Thanks again
yaya
2nd question: I do have the mean and standard deviation of the immediate and delayed recall, is that what you are asking? However, I am not looking at the difference between these two variables, I'm using them to calculate percent of material recalled at delay  mean score on delayed recall divided by mean score on immediate recall x 100. Previously, I was looking at the difference between these two variables, but I ran in too many floor effects so I changed the variable to "percent retention." So, there is not a way to estimate SD in this case other than using what is found in the literature, right?
Thanks again
yaya