Page 1 of 1

MICOM: variances invariance established but not means invariance

Posted: Fri Nov 02, 2018 11:33 am
by Piddzilla
I have two data sets, based on the same survey questions but collected with one year apart. The first set contains 1249 cases and the second set contains 875 cases. I have performed a MICOM analysis to see whether pooling the two data sets is possible.

Configural invariance (step 1) has been established and after running a permutation test in SmartPLS 3, partial compositional invariance (step 2) could also be established. However, concerning step 3, testing whether full measurement invariance holds, I get statistically significant p-values for means but not for variances. That is, if means were the sole focus for this test, full measurement invariance would not have been established. If, on the other hand, variances were the sole focus for this test, full measurement invariance would have been established.

Since the litterature states that both means and variances should be equal for full measurement invariance to be established, I interpret my results as supporting that partial but not full compositional measurement invariance has been established. Thus, multigroup analyses are feasible while pooling data is not.

But what does it mean when the third step of the MICOM test results in contradicting p-values for means and variances? Is this common? Should this be interpreted as a different kind of "failed step 3" compared to when both means and variances produce significant p-values?

[EDIT] I forgot to mention that in my PLS path model, I make use of higher order constructs, measured with latent variable scores. In a comparison of the results from a permutation test of constructs based on manifest indicator variables with the results from a test based on latent variable scores, it appears as if the latent variable score based constructs perform worse in terms of achieved full measurement invariance. My interpretation is that using latent variable scores will mess up the means invariance but not the variance invariance. Or am I reaching?

Re: MICOM: variances invariance established but not means invariance

Posted: Sat Nov 03, 2018 3:35 pm
by jmbecker
1) What is the meaning of a significant mean difference in the MICOM?
It means that the respondent in the one sample have a consistently higher and in the other a consistently lower value on that scale. For example, if the construct would be customer satisfaction, the customers in the one year are more satisfied than in the other year. That could have systematic reasons (e.g., a change in service or product quality) or sampling reasons (e.g., you might have collected more satisfied customer by chance or by incorrectly sampling from the population).

2) Is a mean difference always a problem for pooling?
I would say that it is not necessarily a problem for pooling. This might be contrary to the literature, but only slightly.
It is certainly a problem, if you cannot rule out sampling reasons. If there has been a systematic shift in behavior (e.g., satisfaction) than it should be captured by the changes in the model. Yet, it might be worth exploring both group models separately to find out more about these differences. You may also want to control for the group differences (year of collection) within your model (i.e., as a dummy, which controls purely for the mean differences and/or as a moderator to find changes in the coefficients due to the year of collection). Whether doing a multigroup analysis or a dummy/moderator approach depends on how many constructs/relation are affected.

3) What is the meaning of a significant variance difference in the MICOM?
It would imply that the respondents in the one group had a much higher variability in their answers than in the other group. For example, very satisfied and very unsatisfied in the one group and either very low, very high, or medium in the other group. Such a behavior can especially be a problem for pooling groups because of the routine standardization of constructs in PLS which would be impacted by very different variances of the variables/constructs.

4) Latent variables in two-stage second-order construct behave differently.
This is not surprising as the latent variables are already standardized (mean zero and variance of one). Thus using these in a second-stage could impact the results of MICOM. I would guess that the results are less trustworthy because of the pre-treatment of the latent variables.

Re: MICOM: variances invariance established but not means invariance

Posted: Sun Nov 04, 2018 3:57 pm
by Piddzilla
Dr. Becker,

Thanks for this very clarifying response!