I have two data sets, based on the same survey questions but collected with one year apart. The first set contains 1249 cases and the second set contains 875 cases. I have performed a MICOM analysis to see whether pooling the two data sets is possible.
Configural invariance (step 1) has been established and after running a permutation test in SmartPLS 3, partial compositional invariance (step 2) could also be established. However, concerning step 3, testing whether full measurement invariance holds, I get statistically significant pvalues for means but not for variances. That is, if means were the sole focus for this test, full measurement invariance would not have been established. If, on the other hand, variances were the sole focus for this test, full measurement invariance would have been established.
Since the litterature states that both means and variances should be equal for full measurement invariance to be established, I interpret my results as supporting that partial but not full compositional measurement invariance has been established. Thus, multigroup analyses are feasible while pooling data is not.
But what does it mean when the third step of the MICOM test results in contradicting pvalues for means and variances? Is this common? Should this be interpreted as a different kind of "failed step 3" compared to when both means and variances produce significant pvalues?
[EDIT] I forgot to mention that in my PLS path model, I make use of higher order constructs, measured with latent variable scores. In a comparison of the results from a permutation test of constructs based on manifest indicator variables with the results from a test based on latent variable scores, it appears as if the latent variable score based constructs perform worse in terms of achieved full measurement invariance. My interpretation is that using latent variable scores will mess up the means invariance but not the variance invariance. Or am I reaching?
MICOM: variances invariance established but not means invariance

 SmartPLS Developer
 Posts: 908
 Joined: Tue Mar 28, 2006 11:09 am
 Real name and title: Dr. JanMichael Becker
Re: MICOM: variances invariance established but not means invariance
1) What is the meaning of a significant mean difference in the MICOM?
It means that the respondent in the one sample have a consistently higher and in the other a consistently lower value on that scale. For example, if the construct would be customer satisfaction, the customers in the one year are more satisfied than in the other year. That could have systematic reasons (e.g., a change in service or product quality) or sampling reasons (e.g., you might have collected more satisfied customer by chance or by incorrectly sampling from the population).
2) Is a mean difference always a problem for pooling?
I would say that it is not necessarily a problem for pooling. This might be contrary to the literature, but only slightly.
It is certainly a problem, if you cannot rule out sampling reasons. If there has been a systematic shift in behavior (e.g., satisfaction) than it should be captured by the changes in the model. Yet, it might be worth exploring both group models separately to find out more about these differences. You may also want to control for the group differences (year of collection) within your model (i.e., as a dummy, which controls purely for the mean differences and/or as a moderator to find changes in the coefficients due to the year of collection). Whether doing a multigroup analysis or a dummy/moderator approach depends on how many constructs/relation are affected.
3) What is the meaning of a significant variance difference in the MICOM?
It would imply that the respondents in the one group had a much higher variability in their answers than in the other group. For example, very satisfied and very unsatisfied in the one group and either very low, very high, or medium in the other group. Such a behavior can especially be a problem for pooling groups because of the routine standardization of constructs in PLS which would be impacted by very different variances of the variables/constructs.
4) Latent variables in twostage secondorder construct behave differently.
This is not surprising as the latent variables are already standardized (mean zero and variance of one). Thus using these in a secondstage could impact the results of MICOM. I would guess that the results are less trustworthy because of the pretreatment of the latent variables.
It means that the respondent in the one sample have a consistently higher and in the other a consistently lower value on that scale. For example, if the construct would be customer satisfaction, the customers in the one year are more satisfied than in the other year. That could have systematic reasons (e.g., a change in service or product quality) or sampling reasons (e.g., you might have collected more satisfied customer by chance or by incorrectly sampling from the population).
2) Is a mean difference always a problem for pooling?
I would say that it is not necessarily a problem for pooling. This might be contrary to the literature, but only slightly.
It is certainly a problem, if you cannot rule out sampling reasons. If there has been a systematic shift in behavior (e.g., satisfaction) than it should be captured by the changes in the model. Yet, it might be worth exploring both group models separately to find out more about these differences. You may also want to control for the group differences (year of collection) within your model (i.e., as a dummy, which controls purely for the mean differences and/or as a moderator to find changes in the coefficients due to the year of collection). Whether doing a multigroup analysis or a dummy/moderator approach depends on how many constructs/relation are affected.
3) What is the meaning of a significant variance difference in the MICOM?
It would imply that the respondents in the one group had a much higher variability in their answers than in the other group. For example, very satisfied and very unsatisfied in the one group and either very low, very high, or medium in the other group. Such a behavior can especially be a problem for pooling groups because of the routine standardization of constructs in PLS which would be impacted by very different variances of the variables/constructs.
4) Latent variables in twostage secondorder construct behave differently.
This is not surprising as the latent variables are already standardized (mean zero and variance of one). Thus using these in a secondstage could impact the results of MICOM. I would guess that the results are less trustworthy because of the pretreatment of the latent variables.
Dr. JanMichael Becker, University of Cologne, SmartPLS Developer
Researchgate: https://www.researchgate.net/profile/Ja ... v=hdr_xprf
GoogleScholar: http://scholar.google.de/citations?user ... AAAJ&hl=de
Researchgate: https://www.researchgate.net/profile/Ja ... v=hdr_xprf
GoogleScholar: http://scholar.google.de/citations?user ... AAAJ&hl=de

 PLS Junior User
 Posts: 7
 Joined: Tue Nov 22, 2016 10:39 am
 Real name and title: Peter Bergwall, Doctoral student
Re: MICOM: variances invariance established but not means invariance
Dr. Becker,
Thanks for this very clarifying response!
Thanks for this very clarifying response!