unclear reviewer's comment on construct validity

Questions about the implementation and application of the PLS-SEM method, that are not related to the usage of the SmartPLS software.
Post Reply
irene eleonora
PLS Junior User
Posts: 10
Joined: Mon Mar 26, 2012 3:06 pm
Real name and title:
Location: Università Bocconi

unclear reviewer's comment on construct validity

Post by irene eleonora »

Hello. I have problems in understanding a reviewer's comment on the tests I employed to check the validity of my measures. The reviewer says: "I am not completely satisfied about the dual validity check presented in the paper (using both SPSS and PLS). My advice is to rely on PLS only". I think the reviewer refers to the fact that I performed Exploratory Factor Analysis (using SPSS) on my items before running the PLS model. My doubt is the following: if I cannot use SPSS (as the reviewer asks), is there a way to perform EFA in PLS? I don't think so, I think PLS can only perform Confirmatory Factor Analysis, am I right? If this is the case, should I run a CFA in PLS before running the PLS algorithm or can I directly run my PLS model and assess the measurement model results?

Thank you very much for your help since now.
Best regards

Irene
christian.nitzl
PLS Expert User
Posts: 248
Joined: Sat Jul 25, 2009 1:34 pm
Real name and title:

Re: unclear reviewer's comment on construct validity

Post by christian.nitzl »

Hi Irene,

I would agree with the reviewer comment. However, I have often seen this mixing of EFA and PLS. Although, reputable researchers advise to avoid mixing results from an explorative factor analysis with a confirmatory factor analysis like PLS-SEM because of the problems of model over-fitting and the capitalization of chance (Hair et al., 2016, Chin, 1998, Marcoulides and Chin, 2013). An explorative approach to generate results must always be cross-validated with new data that validity of the findings of an EFA can be claimed (Marcoulides and Chin, 2013). Furthermore, differentiation has to be made between reflective and formative measured constructs. An explorative factor analysis would assume reflective measurements for the separate factors. Mixing these two kinds of measurements can extremely bias the results.

Why do you need a EFA? When you use measurements from other studies I would test the measurements for reliability and validity in PLS directly (e.g., how are the loadings, how is AVE). Otherwise, when you develop your own measurements you need more than one step (cf. Churchill, 1979). That mean also you need more than one survey to develop them.

However, there is no CFA like in CB-SEM in PLS-SEM. In PLS-SEM you build the complete model including the inner path model. In the first step, you look only in the outer model and check the measurements. This can be seen as a kind of CFA in my opinion. Only when the measurements fulfill the quality criteria you check the inner model. Otherwise you have to delete the bad items and calculate the model new.

Hope that helps,

Christian
irene eleonora
PLS Junior User
Posts: 10
Joined: Mon Mar 26, 2012 3:06 pm
Real name and title:
Location: Università Bocconi

Re: unclear reviewer's comment on construct validity

Post by irene eleonora »

Dear Christian,

thank you so much for your exhaustive and useful reply, it helps a lot!

If I do not bother you too much, I would like to share with you another doubt concerning another comment from the same reviewer. With reference to the table in which I report the LV correlations and the Square Root of AVE statistics (as a test of discriminant validity), the reviewer says "the correlations in Table 5 are not flagged with statistical significance. Please amend the table and provide some comments in the text. I would also suggest to use the factor scores instead of the averages and report in a footnote about (hopefully) consistent results".
I have a problem in understanding what the reviewer means when he suggests "use the factor scores instead of the averages". In fact, these correlations are computed by using the LV scores calculated in PLS, and as such I think they are already based on "factor scores" (if I am right, LV scores in PLS are computed by using the outer weights as in a weighted average, and therefore LV scores computed in PLS can be seen as kind of factor scores). Maybe the reviewer was confused because, in a previous table with the decriptive statistics of my variables, I wrote that such descritpive statistics are based on the average scores of the multi-item constructs.

Do you think my understandng is correct? In other words, can I answer to the reviewer that the correlations are indeed already based on "factor scores" (and not on "averages") as they are based on the standardized LV scores computed in PLS?

Sorry for the lenghty question, but I wanted to be sure to clearly express my doubt. I hope you can understand and, hopefully, reply (a simple "yes, you are right" or "not, you are not right" will be more than enough!)
Never ending thanks and best wishes

Irene
christian.nitzl
PLS Expert User
Posts: 248
Joined: Sat Jul 25, 2009 1:34 pm
Real name and title:

Re: unclear reviewer's comment on construct validity

Post by christian.nitzl »

Dear Irene,

I am not fully sure however it really seems that the reviewer was irritated about your mentioned point. It is exactly as you assume that the correlations are based on the weighted factor scores in PLS.

What is disturbing me a little bit is that the reviewer seems to expect that you should test the correlation for significance. However, in my opinion that is somewhat critical because testing it means you assume a normal distribution what is contradicting the assumption of PLS as a distribution free approach.

Much more reliable are the new criteria HTMT for testing the discriminant validity. It is already implemented in the new SmartPLS 3. I would highly recommend you to use this criteria if you do not use it already.

Best regards,
Christian

P.S. Please say hi to Angelo D. at your Bocconi university if you should see him ;-)
irene eleonora
PLS Junior User
Posts: 10
Joined: Mon Mar 26, 2012 3:06 pm
Real name and title:
Location: Università Bocconi

Re: unclear reviewer's comment on construct validity

Post by irene eleonora »

Dear Christian,

thank you once again for your reply and your useful suggestion on SmartPLS 3!
Of course, I will say hello to Angelo on your behalf as soon as I see him!!!

Best wishes

Irene
jmbecker
SmartPLS Developer
Posts: 1282
Joined: Tue Mar 28, 2006 11:09 am
Real name and title: Dr. Jan-Michael Becker

Re: unclear reviewer's comment on construct validity

Post by jmbecker »

Adding to the aforementioned things, I would reconsider reporting the descriptive statistics based on the average scores of the multi-item constructs. Instead, you should also report the descriptives based on the PLS latent variable scores. You will get unstandardized scores in the IPMA procedure, from which you can calculate means and standard deviations, etc.
Dr. Jan-Michael Becker, BI Norwegian Business School, SmartPLS Developer
Researchgate: https://www.researchgate.net/profile/Jan_Michael_Becker
GoogleScholar: http://scholar.google.de/citations?user ... AAAJ&hl=de
irene eleonora
PLS Junior User
Posts: 10
Joined: Mon Mar 26, 2012 3:06 pm
Real name and title:
Location: Università Bocconi

Re: unclear reviewer's comment on construct validity

Post by irene eleonora »

Dear Jan-Michael,
thank you very much for your nice suggestion, you are perfectly right, reporting the decriptive statistics based on average scores is definitely misleading.

Best wishes

Irene
Post Reply