Some questions about HTMT and cross-loadings

Questions about the implementation and application of the PLS-SEM method, that are not related to the usage of the SmartPLS software.
Post Reply
agalvez
PLS Expert User
Posts: 39
Joined: Mon Jul 04, 2016 10:17 pm
Real name and title: Alex

Some questions about HTMT and cross-loadings

Post by agalvez »

Hi mates

I would be very greatfull if you could help me with the following questions:

I just received a revision of my paper. One of the reviewers said that correlations among the LVs in my model are high (about 0.60) and may be multicollinearity problems. I think some LVs are conceptually very similar and this is the reason of high correlations.

It's important to mention that in the previous version of the paper I didn't report VIF values nor said the model was free of multicollinearity problems. I think that if a reviewer see that some correlations are above 0.60, and you didn't mention collinearity was checked, he thinks this may be the cause of high correlations.

I checked my model again.

1) Discriminant Validity. In the previous version I applied Fornell-Larcker. Now, I report HTMT as recommended. Discriminant validity is not confirmed between one IV and DV. In the revised version of the paper I will report HTMT.

In order to fix the discriminant validity between IV and DV, I checked the cross-loadings. As expected, items belonging to IV also load on DV. I decided to remove the problematic IV from the model. I didn't check cross-loadings in the previous version of the paper, given that discriminant validity was confirmed with Fornell-Larcker criterion. However, some cross-loadings values are greater than 0.50 (see 2). This is my main concern.

2) Also, in cross-loadings matrix I observed that cross-loadings are between 0.60 and 0.70. Is this a problem? I read that threshould should be 0.50. However, if I establish 0.50, many of items loads also in other factor.

Other researchers argue there is not a threshold and that if an item loads more in their construct than in other, then, discriminant validity is confirmed. Some researchers say that should be a difference of, at least, 0.20 between item-factor loadings and cross-loadings. Is this enought? My model meets this.

On the other hand, should I concern about this, given that all HTMT values are ok? Also, all inner VIFs are under 5 (some even < 3).

3) In line with 2), when checking cross-loadings, you look at the cross-loadings of the LVs in the model. That is to say, it doesn't matter if a variable is predictor or dependent, you have to check that all cross-loadings are ok. Is this correct?

4) Finally, if factor loadings, CR, AVE, VIF, HTMT values are ok... But correlations among LVs are about 0.60... what could I do? I think the only way is to remove high-correlated items, but I dont understand this since the other values are ok. On the other hand, it is expected that conceptually similar LVs to be correlated. Imagine Satisfaction and Loyalty. Also, in "Primer on partial least squares" book I read that, in marketing studies, R2 may exceed 0.75. I think this is because variables are very similar (may be correlated, as in my study).

Taking all the above into account, do you think my model is ok or should I check something else?

Thank you in advance
jmbecker
SmartPLS Developer
Posts: 1282
Joined: Tue Mar 28, 2006 11:09 am
Real name and title: Dr. Jan-Michael Becker

Re: Some questions about HTMT and cross-loadings

Post by jmbecker »

I think you already mentioned most important aspects.

If constructs are conceptually similar, it is expected that they correlate highly. Discuss this in your report.
Most importantly, it is necessary to assure discriminant validity between construct. A high correlation is not necessarily a problem if discriminant validity is established. If discriminant validity is not established the constructs might be conceptually to similar (or not well measured) and therefore conflate the results. I would focus on HTMT as it is the most valid index of discriminant validity. If HTMT is ok, then this should be a strong argument for your model.
Cross-loadings are problematic as they often do not perform well in detecting discriminant validity problems. They might be good to detect which specific indicator could be the reason for a failure of HTMT and thus for indicator removal (or repositioning), but I would generally not base to much weight on this assessment.

Finally, assess multicollinearity. If this is below common thresholds it is also an argument for your model. Also, if there are no strange results, collinearity is often not a big problem. It only becomes a problem if there are suppression effects, sign reversals, etc.

Of course, all other construct indices should be good as well (AVE, CR, etc.).
Dr. Jan-Michael Becker, BI Norwegian Business School, SmartPLS Developer
Researchgate: https://www.researchgate.net/profile/Jan_Michael_Becker
GoogleScholar: http://scholar.google.de/citations?user ... AAAJ&hl=de
agalvez
PLS Expert User
Posts: 39
Joined: Mon Jul 04, 2016 10:17 pm
Real name and title: Alex

Re: Some questions about HTMT and cross-loadings

Post by agalvez »

jmbecker wrote:I think you already mentioned most important aspects.

If constructs are conceptually similar, it is expected that they correlate highly. Discuss this in your report.
Most importantly, it is necessary to assure discriminant validity between construct. A high correlation is not necessarily a problem if discriminant validity is established. If discriminant validity is not established the constructs might be conceptually to similar (or not well measured) and therefore conflate the results. I would focus on HTMT as it is the most valid index of discriminant validity. If HTMT is ok, then this should be a strong argument for your model.
Cross-loadings are problematic as they often do not perform well in detecting discriminant validity problems. They might be good to detect which specific indicator could be the reason for a failure of HTMT and thus for indicator removal (or repositioning), but I would generally not base to much weight on this assessment.

Finally, assess multicollinearity. If this is below common thresholds it is also an argument for your model. Also, if there are no strange results, collinearity is often not a big problem. It only becomes a problem if there are suppression effects, sign reversals, etc.

Of course, all other construct indices should be good as well (AVE, CR, etc.).
Prof. Becker, I greatly appreciate your help.

Now I have clear that I should check discriminant validity with HTMT approach. And you also provided me some guidelines to answer the reviewers. Thank you.

If you let me, I have two additional questions regarding this model about indirect effects. I have the following two situations:

First

Direct effect: -0.02 (t-value < 1.65) --> non-significant
Indirect effect: 0.15 (t-value > 2.57) --> significant
Total effect: Direct + Indirect = -0.02 + 0.15 = 0.13 (t-value > 2.57) --> significant

If the direct effect is non-significant, should not the total effect be equal to the indirect effect? That is to say, 0.15.

Supposing that the above values are not strange, could I say... IV has no direct effect on DV (-0.02; t-value<1.65), but there is a significant indirect effect through MV (0.15; t-value>2.57). Thus, the total effect of IV on DV is 0.13 (t-value>2.57).

Second

Direct effect: -0.086 (t-value < 1.65) --> non-significant
Indirect effect: 0.128 (t-value > 2.57) --> significant
Total effect: Direct + Indirect = -0.086 + 0.128 = 0.042 (t-value < 1.65) --> non-significant

How do you interpret this? If my goal is to study the total effect of X on Y, should I say that there is no total effect even if there is a positive and significative indirect effect? Should I simply consider that there is no total effect of X on Y? Is this called "inconsistent mediation?

Thank you
jmbecker
SmartPLS Developer
Posts: 1282
Joined: Tue Mar 28, 2006 11:09 am
Real name and title: Dr. Jan-Michael Becker

Re: Some questions about HTMT and cross-loadings

Post by jmbecker »

Your results are possible. The total effect is the indirect effect + the direct effect. If you have slightly negative effects for the direct effect a positive indirect effect might be reduced to a smaller total effect. This total effect can then be significant or insignificant depending on much the (insignificant) direct effect reduces the total effect.

If you are interested in a mediation, I would focus not so much on the total effect, but only on the indirect and direct effect (full vs. partial mediation; in your case full mediation). There could be other possible mediators (with same or different sign) that shape the total effect.
Dr. Jan-Michael Becker, BI Norwegian Business School, SmartPLS Developer
Researchgate: https://www.researchgate.net/profile/Jan_Michael_Becker
GoogleScholar: http://scholar.google.de/citations?user ... AAAJ&hl=de
agalvez
PLS Expert User
Posts: 39
Joined: Mon Jul 04, 2016 10:17 pm
Real name and title: Alex

Re: Some questions about HTMT and cross-loadings

Post by agalvez »

jmbecker wrote:Your results are possible. The total effect is the indirect effect + the direct effect. If you have slightly negative effects for the direct effect a positive indirect effect might be reduced to a smaller total effect. This total effect can then be significant or insignificant depending on much the (insignificant) direct effect reduces the total effect.

If you are interested in a mediation, I would focus not so much on the total effect, but only on the indirect and direct effect (full vs. partial mediation; in your case full mediation). There could be other possible mediators (with same or different sign) that shape the total effect.
Dear jmbecker

Thank you for your answer.

A reviewer told me that the main reason to study indirect effects is to determine whether total effects are significant.

My model has 5 IV, 1 MV and 1 DV. All the constructs are linked. The following is an example of the relation between one IV, MV and DV:

Perceived usefulness (PU) --> Satisfaction (SA) --> Loyalty (LO)

(imagine four more predictors)

I'm studying the effects of these predictors on Satisfaction and also, throught it, on Satisfaction. For Satisfaction, I check the direct effects. For Loyalty, I check the direct and indirect effects, in order to conclude if the total effect is either significant or not.

What happens in ONE of the cases is that the direct effect is significant, indirect is not and, therefore, total effect can be either significant or not. As you commented, it depends on the direction and strenght of direct and indirect effects. Should I conclude reporting the significance of the total effect?

On the other hand, I dont know what do you mean with "there could be other possible mediators that shape the total effect". Do you refer to more mediators in my model (it only has one) or "missing" mediators (not present in my study) that, if they were taken into account, could affect my total effect?

Thank you in advance.
User avatar
cringle
SmartPLS Developer
Posts: 818
Joined: Tue Sep 20, 2005 9:13 am
Real name and title: Prof. Dr. Christian M. Ringle
Location: Hamburg (Germany)
Contact:

Re: Some questions about HTMT and cross-loadings

Post by cringle »

You can also directly take a look at the total effects. The bootstrap results of the total effects allow you to determine if they are significant.

Best regards
CR
agalvez
PLS Expert User
Posts: 39
Joined: Mon Jul 04, 2016 10:17 pm
Real name and title: Alex

Re: Some questions about HTMT and cross-loadings

Post by agalvez »

Thank you Dr. Ringle
Post Reply