q² effect size

Frequently asked questions about PLS path modeling.
Post Reply
imdad7371
PLS Junior User
Posts: 2
Joined: Tue Jan 16, 2018 9:39 am
Real name and title: Mr. Imdadulah

Re: Some methodological issues

Post by imdad7371 » Tue Jan 16, 2018 9:41 am

When assessing q², I get negative values. Can I ignore the negative symbol and interpret the magnitude of the value whether positive or negative?

jmbecker
SmartPLS Developer
Posts: 820
Joined: Tue Mar 28, 2006 11:09 am
Real name and title: Dr. Jan-Michael Becker

Re: Some methodological issues

Post by jmbecker » Tue Jan 16, 2018 10:35 am

Generally, q² can also be negative unlike f².
The R² (i.e., explained variance) always improves if you add more predictors (and therefore f² is always positive).
The predictive relevance (Q²) of a model can also decrease if you add predictors, due to overfitting of the model (e.g., adding irrelevant noisy predictors).
Thus, theoretically, a negative q² means that removing the focal variable actually improves the predictive power of the model (Q²). That is, in terms of predictions, you are better off without the focal variable.
However, there are some potential concerns with calculating q² in a PLS model (which are also the reason why we have not implemented such a calculation).
For example, by deleting a latent variable you are changing the model and thus the PLS estimates might change as they are dependent on the structural model. Whether this is good or not (correct or not) when calculating q² has never been finally decided.
Deleting variables from the PLS model may change the order of manifest variables in the missing value substitution and prediction process. This may positively or negatively effect the predictive relevance without having a reason in the deletion of the focal variable.
Therefore, the change in Q² may not be attributable only to the inclusion/exclusion of the focal variable. Thus, q² might not be a reliable measure, which is why I usually do not recommend it.
At the moment, more research is necessary to more fully understand this type of analysis and how it can be correctly applied.
Dr. Jan-Michael Becker, University of Cologne, SmartPLS Developer
Researchgate: https://www.researchgate.net/profile/Ja ... v=hdr_xprf
GoogleScholar: http://scholar.google.de/citations?user ... AAAJ&hl=de

MichaelMug
PLS Junior User
Posts: 1
Joined: Mon Jul 09, 2018 1:50 am
Real name and title: MichaelMug

q effect size

Post by MichaelMug » Tue Jul 24, 2018 11:51 pm

Hey Dev, thanks for getting back to me. This is actually a z-statistic, along with a p-value. I'm not sure what the relationship is to a standardized z-score?? Maybe I didn't state the question correctly So does that change anything regarding making it possible to convert to an effect size? I know if you have an F-statistic with a p-value, there is a method to convert it with your software, for example to an effect size without having the means and SD's -- was wondering if there is anything similar for a z-statistic?? I've been searching online and reading all my old stats books but can't come up with anything.

2nd question: I do have the mean and standard deviation of the immediate and delayed recall, is that what you are asking? However, I am not looking at the difference between these two variables, I'm using them to calculate percent of material recalled at delay - mean score on delayed recall divided by mean score on immediate recall x 100. Previously, I was looking at the difference between these two variables, but I ran in too many floor effects so I changed the variable to "percent retention." So, there is not a way to estimate SD in this case other than using what is found in the literature, right?

Thanks again

ya-ya

Post Reply