effect size

Frequently Asked Questions about the SmartPLS software. Please refer to this section first, if you have problems with the software.
davidal
PLS Senior User
Posts: 21
Joined: Tue May 27, 2008 4:19 pm
Real name and title:

effect size

Post by davidal » Wed Jan 13, 2010 4:21 pm

Dear all,

I have a beginner question concerninr effect size (f2).

Based on the formula f2 is computed as follows : (R2included - R2 excluded)/(1 - R2 included)

Imagine I have a paht model with
A-->C-->D
B-->C-->D

In this model, it is possible to apply the formula for C: A is excluded and the new R2 of C is calculated; then B is excluded and another R2 of C is calculated. As a result we get 3 R2: one with A and B; one with A only; one with B only.

But how is it possible to apply this formula when there is only one exogenous variable (as is the case for the path C-->D). I cannot exclude C otherwise the model is not complete. Is it correct to pretend that R2 for D=0 when C is excluded ? Or is there another solution?

Thank you very much for your help.

David

User avatar
Diogenes
PLS Super-Expert
Posts: 905
Joined: Sat Oct 15, 2005 5:13 pm
Real name and title:
Location: São Paulo - BRAZIL
Contact:

Post by Diogenes » Thu Jan 14, 2010 12:10 am

Hi David,

See
COHEN, Jacob. (1977). Statistical Power Analysis for the Behavioral Sciences. Revised Edition. New York: Academic Press.

In the chapter 9 (F Tests of Variance Proportions in Multiple Regression / Correlation Analysis (p.407-414), we have:

R2 = f2 / (1 + f2)

f2 = R2 / (1 - R2)

where
f2 = 0.02 is small
f2 = 0.15 is medium
f2 = 0.35 is large

Best regards,

Bido

davidal
PLS Senior User
Posts: 21
Joined: Tue May 27, 2008 4:19 pm
Real name and title:

Post by davidal » Thu Jan 14, 2010 9:04 am

Dear Bido,

Thank you for your answer. But I'm not sure we're talking about the same thing. What I want to estimate is the f2 for each effect in the model in order to gauge the effect of each predictor in the model.
To that purpose, Henseler et al. 2008 suggested to use this formula:
f2 = (R2incl-R2excl)/(1-R2incl).
My problem is the application of this formula when there is only one predictor. Maybe your formula must then be used?

Thanks for answers.

David

User avatar
Diogenes
PLS Super-Expert
Posts: 905
Joined: Sat Oct 15, 2005 5:13 pm
Real name and title:
Location: São Paulo - BRAZIL
Contact:

Post by Diogenes » Fri Jan 15, 2010 6:26 pm

Hi David,

I looked for more information, and…

1) f2 = R2 / (1 - R2) --> used in multiple regression considering all independent variables (It is not what you are trying to assess).

2) (R2included - R2 excluded)/(1 - R2 included) --> used in multiple regression considering the partial coefficients (ok for A and B variables, ok as you have done).

3) Eventually, as you have just one relation ( C --> D ) the beta = r, then, the effect size will be the value of this correlation (Cohen, 1977, p.75-80), with:
r = 0.10 = small effect
r = 0.30 = medium effect
r = 0.50 = large effect

I hope this help.

Bido

elianacarraca
PLS User
Posts: 13
Joined: Mon Oct 13, 2008 5:30 pm
Real name and title:

Post by elianacarraca » Tue May 18, 2010 6:40 pm

Hi!

I have a similar model A -> C -> D
B -> C -> D

But what I would like to know is if A is a stronger predictor then B, and if A has a greater contribution to the R2 (variance explained) of the endogenous variables, when compared to B.

I remembered I could test the model with and without A and compare both the R2 for C (and also D), as well as the f2 (effect sizes). Is this possible? Is it correct? How can I test for the R2-change significance? I believe it will be necessary. Do you agree?

I read something in this forum suggesting we could calculate it in spss, but I don't know how... should I compute and use global scores for each of my LV in the model?

I still have one more doubt... I realized that the R2 for D (the late endogenous variable, not directly linked to A or B) did not change with the inclusion or exclusion of A from the model. Why does this happen? Is it related with the partial nature of the analysis performed in PLS?

I appreciate it enormously if professor Bido or anyone could reply to this post and clarify my doubts.

Thanks in advance!

Eliana Carraça

christian.nitzl
PLS Expert User
Posts: 248
Joined: Sat Jul 25, 2009 1:34 pm
Real name and title:

Post by christian.nitzl » Fri Jun 18, 2010 3:47 pm

Hey Eliana,

Here comes my opinion about the use of effect size in your model. Maybe it will help you:

a) I would use an excel sheet to calculate the effect size. As you described above you calculate one time your model with and the other time without A (or B). Then you put the different R2 in the formula of Cohen (R2 incl.- R2 exc. / 1-R2 incl). Thereafter you can compare the different effect sizes.

b) I would not use a F-test, because “it is more convenient to work directly with f2 rather than f.” (Cohen, 1988, p. 480) Normally you use the F-test if you are examining simultaneously the impact of “sets of predictors” for a dependent construct.

c) The R2 for D does not change, because the independent variable A has no impact on it.

Best regards

Christian

stephan.kramer
PLS Expert User
Posts: 37
Joined: Tue Mar 25, 2008 9:16 am
Real name and title:

Post by stephan.kramer » Mon Jun 21, 2010 10:10 am

Hi Elina,
christian.nitzl wrote: c) The R2 for D does not change, because the independent variable A has no impact on it.
or to be a bit more precise, it doesn't change because PLS uses a blockwise estimation procedure. Thus, if the variables are not connected, no changes in R² occur if you omit the independent variable.

Best, Stephan

elianacarraca
PLS User
Posts: 13
Joined: Mon Oct 13, 2008 5:30 pm
Real name and title:

Post by elianacarraca » Mon Jun 21, 2010 10:17 am

Thanks to both for your answers!!

Eliana

christian.nitzl
PLS Expert User
Posts: 248
Joined: Sat Jul 25, 2009 1:34 pm
Real name and title:

Post by christian.nitzl » Mon Jun 21, 2010 10:38 am

Hey Stephan,

Thanks for your comment.

I test such a case in my model, like mentioned from Elian (e.g. A-B-C with no direct connection from A to C).

However R2 has been changed for my model, but in a very small way (0.01).

One reason, why this is so, I can see in the iteration process of pls. The LV scores aren't fixed in the first iteration. At the beginning you calculate your LV scores than the connections between the LV variables [for ease]. This has again an impact on the estimated LV before and so on.

But something else came in my mind: perhaps the "total effect" can be giving Eliana some more information about the impact of variable A and the test of significance of it.

Best regards

Christian
Last edited by christian.nitzl on Fri Jun 25, 2010 12:46 pm, edited 2 times in total.

stephan.kramer
PLS Expert User
Posts: 37
Joined: Tue Mar 25, 2008 9:16 am
Real name and title:

Post by stephan.kramer » Tue Jun 22, 2010 7:46 pm

Hi Christian,

I think the explanation for this phenomenon is that the PLS-algorithm switches between the estimate for the measurement model and the structural model while iterating. If I understood the algorithm correctly, it converges to a solution when the differences between the inner and the outer approximation fall below a certain (very small) threshold. Thus, unlike covariance-based techniques, the estimation of the measurement and the structural model is NOT independent from each other.

The consequence of this fact is that as soon as you omit an IV (like you did in your model), slight changes in the measurement model of the mediating variable can occur which finally lead to an (likewise small) alteration of the path coefficient to the DV. So while it is still correct that the structural model by itself is estimated blockwise, it is possible that minor changes occur due to an alteration of the measurement model of the mediating variable.

At least this would be my explanation for this phenomenon. Any comments are appreciated.

Best, Stephan

elianacarraca
PLS User
Posts: 13
Joined: Mon Oct 13, 2008 5:30 pm
Real name and title:

Post by elianacarraca » Mon Jul 12, 2010 2:10 pm

davidal wrote:Dear Bido,

Thank you for your answer. But I'm not sure we're talking about the same thing. What I want to estimate is the f2 for each effect in the model in order to gauge the effect of each predictor in the model.
To that purpose, Henseler et al. 2008 suggested to use this formula:
f2 = (R2incl-R2excl)/(1-R2incl).
My problem is the application of this formula when there is only one predictor. Maybe your formula must then be used?

Thanks for answers.

David

Is there a way of testing the significance of the difference between two paths leading to the same DV? That is, A -> C and B -> C
How can I say that one effect is significantly stronger than the other? Is there a way of testing this with PLS?


Can calculating the effect size help me...? f2 = (R2incl-R2excl)/(1-R2incl).
And if so, how could I do it? By applying the formula for C excluding A and obtaining a new R2 of C; then excluding B and obtaining another R2 of C. And finally, including A and B and obtaining a final R2, and then calculating both f2 and comparing them?

Thanks!

Eliana

rayouby
PLS Junior User
Posts: 5
Joined: Thu Jul 26, 2007 2:07 am
Real name and title:

Overall Effect Size

Post by rayouby » Tue Oct 19, 2010 3:47 pm

Hello Everyone,

Thank you for the detailed discussion! I want to make sure I am applying the information correctly.

I need to calculate the Overall Effect Size of my model. My model is in the following form:

A-->B1-->C
A-->B2-->C
A-->B3-->C

I have three questions:

1. Does using the formula given below for f2 produce what is referred to as the "Overall Effect Size"?

f2 = R2 / (1 - R2)

2. I assume that I would use the R2 of C for this calculation. Is this correct?

3. My R2 is 0.457 and thus my f2 is 0.842 . According to the guidelines below, my f2 is huge! Is this possible?

Thank you very much!
Reem


Professor Bido wrote:
See
COHEN, Jacob. (1977). Statistical Power Analysis for the Behavioral Sciences. Revised Edition. New York: Academic Press.

In the chapter 9 (F Tests of Variance Proportions in Multiple Regression / Correlation Analysis (p.407-414), we have:

R2 = f2 / (1 + f2)

f2 = R2 / (1 - R2)

where
f2 = 0.02 is small
f2 = 0.15 is medium
f2 = 0.35 is large
Reem Ayouby, M.Sc.
Ph.D. Student
JMSB, Concordia University
Reem.Ayouby@gmail.com
r_ayouby@jmsb.concordia.ca

christian.nitzl
PLS Expert User
Posts: 248
Joined: Sat Jul 25, 2009 1:34 pm
Real name and title:

Post by christian.nitzl » Tue Oct 19, 2010 4:22 pm

Hey Reem,

For me it makes no sense to use the cited formula f2=R2/(1-R2) if you use PLS. This formula would be useful only, as Prof. Diogenes wrote, for multiple regression (or the special case of David at the beginning of this discussion). That is the case if you have only direct effects on ‘one’ variable. In this case you should use a multiple regression. As far as I understand it, this is not the case in your model. Therefore you should use the formula f2 = (R2incl-R2excl)/(1-R2incl). Then you can calculate R2 with and without e.g. B1 and put this values in the formula. As Stephan and I discussed above, it would be no difference (or only small difference) if you use A for your calculation, because A has no direct connection to C.

Best regards,

Christian

rayouby
PLS Junior User
Posts: 5
Joined: Thu Jul 26, 2007 2:07 am
Real name and title:

Overall Effect Size in PLS

Post by rayouby » Tue Oct 19, 2010 6:38 pm

Hi Christian,

Thank you for your reply!

The reason I think the formula f2 = (R2incl-R2excl)/(1-R2incl) does not apply to my case is that I don't want to know the effect of including one of the variables--or not including it; I simply want to know the overall effect size of the whole model. I Should add that my model is where A is an independent variable, B1, B2, and B3 are mediating variables and C is the dependent variable. All are included in the model simultaneously.

1. In my case, would the formula proposed by Dr. Bido at the beginning--f2=R2/(1-R2)-- apply? From what you say it seems that it would not apply for a PLS model.

2. So, which formula/method does one have to use to calculate the overall effect size for a PLS model (with all variables included)?

Looking forward to hearing what you think about this!
Reem
Reem Ayouby, M.Sc.
Ph.D. Student
JMSB, Concordia University
Reem.Ayouby@gmail.com
r_ayouby@jmsb.concordia.ca

christian.nitzl
PLS Expert User
Posts: 248
Joined: Sat Jul 25, 2009 1:34 pm
Real name and title:

Post by christian.nitzl » Wed Oct 20, 2010 8:14 am

Hey Reem,

ad 1) In my opinion R2 by itself is enough. Because R2 is the percentage of the explained variance of your focal variable C through your model. In other words your whole “effect” . Why do you need the effect size for your model if you won’t test the separate effects of the mediating variables?

ad 2) Perhaps the Stone-Geisser-Test is what are you searching for. It is testing how useful your model is for prediction. That is an other way for measure the “impact” of your model.

I hope this will help you!

Best Regards,

Christian

Post Reply