Dear everybody,
I am doing MICOM procedure of MGA (Multigroup analysis) for more than three groups. Due to a recommendation from a paper, the setting specifications: p value should be made adjustment (Bonferroni corrections): 0.05 to 0.0169524 when executing the permutation test.
However, in Permutation Multigroup Analysis of SmartPLS 4, the significance level automatically changes from 0.017 to 0.02.
Similarly, When I conduct Bootstrap multigroup analysis in Smartpls 4, the significance level automatically changes from 0.017 to 0.02.
How can I deal with this problem?
The link of paper recommending p value adjustment: https://www.sciencedirect.com/science/a ... 6322010049
I wish you a good working day ahead,
Best regard,
Hung
pvalue adjustment: 0.05 to 0.0169524 when executing the permutation test

 PLS Junior User
 Posts: 7
 Joined: Sun Sep 10, 2023 1:16 pm
 Real name and title: Mr. Hung
Re: pvalue adjustment: 0.05 to 0.0169524 when executing the permutation test
Hi everyone,Capcap1668 wrote: ↑Sun Sep 24, 2023 12:58 pm Dear everybody,
I am doing MICOM procedure of MGA (Multigroup analysis) for more than three groups. Due to a recommendation from a paper, the setting specifications: p value should be made adjustment (Bonferroni corrections): 0.05 to 0.0169524 when executing the permutation test.
However, in Permutation Multigroup Analysis of SmartPLS 4, the significance level automatically changes from 0.017 to 0.02.
Similarly, When I conduct Bootstrap multigroup analysis in Smartpls 4, the significance level automatically changes from 0.017 to 0.02.
How can I deal with this problem?
The link of paper recommending p value adjustment: https://www.sciencedirect.com/science/a ... 6322010049
I wish you a good working day ahead,
Best regard,
Hung
Thank you Hung for posting this question. I am also facing the same problem. Appreciate any help on how to handle these issues?
Regards
Dasun

 SmartPLS Developer
 Posts: 1267
 Joined: Tue Mar 28, 2006 11:09 am
 Real name and title: Dr. JanMichael Becker
Re: pvalue adjustment: 0.05 to 0.0169524 when executing the permutation test
To be honest, I don't understand what you mean with "the significance level automatically changes from 0.017 to 0.02."
If you run a (preferably permutationbased MGA), you will get a pvalue for the difference in parameter between groups. You usually conclude that the difference is significant (whether significance this is a meaningful concept in its own is an entirely different question) when the pvalue is below a specific threshold (usually 0.05, which represents your tolerable type I error rate). In the case of multiple pairwise group comparison you need to adjust for the inflation in error rate and lower the threshold (i.e., the 0.05). Thus, you compare your pvalue against a lower threshold. The difference may then not be significant anymore.
Example:
The pvalue for the difference in parameter A between group 1 and 2 is 0.03.
If you compare only the two groups you may conclude significance because 0.03 is smaller than 0.05 (i.e., it is significant on a 5% error level).
If you compare three groups you may not conclude significance because 0.03 is larger than 0.05/3=0.0167 (i.e., it is not significant on a 5% error level accounting for the inflation in error rate due to multiple group comparisons using Bonferoni).
The pvalue for the difference in parameter B between group 1 and 2 is 0.006.
If you compare only the two groups you may conclude significance because 0.006 is smaller than 0.05 (i.e., it is significant on a 5% error level).
If you compare three groups you may also conclude significance because 0.006 is smaller than 0.05/3=0.0167 (i.e., it is significant on a 5% error level accounting for the inflation in error rate due to multiple group comparisons using Bonferoni).
If you run a (preferably permutationbased MGA), you will get a pvalue for the difference in parameter between groups. You usually conclude that the difference is significant (whether significance this is a meaningful concept in its own is an entirely different question) when the pvalue is below a specific threshold (usually 0.05, which represents your tolerable type I error rate). In the case of multiple pairwise group comparison you need to adjust for the inflation in error rate and lower the threshold (i.e., the 0.05). Thus, you compare your pvalue against a lower threshold. The difference may then not be significant anymore.
Example:
The pvalue for the difference in parameter A between group 1 and 2 is 0.03.
If you compare only the two groups you may conclude significance because 0.03 is smaller than 0.05 (i.e., it is significant on a 5% error level).
If you compare three groups you may not conclude significance because 0.03 is larger than 0.05/3=0.0167 (i.e., it is not significant on a 5% error level accounting for the inflation in error rate due to multiple group comparisons using Bonferoni).
The pvalue for the difference in parameter B between group 1 and 2 is 0.006.
If you compare only the two groups you may conclude significance because 0.006 is smaller than 0.05 (i.e., it is significant on a 5% error level).
If you compare three groups you may also conclude significance because 0.006 is smaller than 0.05/3=0.0167 (i.e., it is significant on a 5% error level accounting for the inflation in error rate due to multiple group comparisons using Bonferoni).
Dr. JanMichael Becker, BI Norwegian Business School, SmartPLS Developer
Researchgate: https://www.researchgate.net/profile/Jan_Michael_Becker
GoogleScholar: http://scholar.google.de/citations?user ... AAAJ&hl=de
Researchgate: https://www.researchgate.net/profile/Jan_Michael_Becker
GoogleScholar: http://scholar.google.de/citations?user ... AAAJ&hl=de
Re: pvalue adjustment: 0.05 to 0.0169524 when executing the permutation test
Dear Dr. Beckerjmbecker wrote: ↑Fri Oct 13, 2023 7:11 am To be honest, I don't understand what you mean with "the significance level automatically changes from 0.017 to 0.02."
If you run a (preferably permutationbased MGA), you will get a pvalue for the difference in parameter between groups. You usually conclude that the difference is significant (whether significance this is a meaningful concept in its own is an entirely different question) when the pvalue is below a specific threshold (usually 0.05, which represents your tolerable type I error rate). In the case of multiple pairwise group comparison you need to adjust for the inflation in error rate and lower the threshold (i.e., the 0.05). Thus, you compare your pvalue against a lower threshold. The difference may then not be significant anymore.
Example:
The pvalue for the difference in parameter A between group 1 and 2 is 0.03.
If you compare only the two groups you may conclude significance because 0.03 is smaller than 0.05 (i.e., it is significant on a 5% error level).
If you compare three groups you may not conclude significance because 0.03 is larger than 0.05/3=0.0167 (i.e., it is not significant on a 5% error level accounting for the inflation in error rate due to multiple group comparisons using Bonferoni).
The pvalue for the difference in parameter B between group 1 and 2 is 0.006.
If you compare only the two groups you may conclude significance because 0.006 is smaller than 0.05 (i.e., it is significant on a 5% error level).
If you compare three groups you may also conclude significance because 0.006 is smaller than 0.05/3=0.0167 (i.e., it is significant on a 5% error level accounting for the inflation in error rate due to multiple group comparisons using Bonferoni).
Thank you for the explanation.
I was under the impression that we need to run the 'Permutation multigroup' procedure by adding the Šidák p value adjustment (0.0167) as the significance level to the procedure. When we did that software automatically round up the value to 0.2. (Please see below screenshot). Cheah et al. (2023) also mentioned about this.
(Cheah et al., 2023, p. 10)In this study, the pvalue adjustment based on the Sid ˇ ak ´ procedure was used to assess the compositional invariance, the composites’ equality of mean values, and variances of the composites. If SmartPLS 3 is used, the adjustment of the permutation test will be in three decimal point that is 0.017. In contrast, if SmartPLS4 is used, the adjustment of the permutation test will be in two decimal point that is 0.02.
What I understood from your answer is that we need to run the MGA with 0.05 as significance level but assess the result using Šidák p value adjustment.
Thank you!
Regards
Dasun
Reference
Cheah, J.H., Amaro, S., & Roldán, J. L. (2023). Multigroup analysis of more than two groups in PLSSEM: A review, illustration, and recommendations. Journal of business research, 156, 113539. https://doi.org/https://doi.org/10.1016 ... 022.113539

 SmartPLS Developer
 Posts: 1267
 Joined: Tue Mar 28, 2006 11:09 am
 Real name and title: Dr. JanMichael Becker
Re: pvalue adjustment: 0.05 to 0.0169524 when executing the permutation test
First, whether you use Sidak or Bonferroni is your choice that you need to justify. I have not seen clear evidence that the one is always superior over the other. I was using Bonferoni, because it simplified the example as it is much simpler to calculate.
What you put in the significance field in SmartPLS 4, affects the confidence intervals and the coloring of the pvalues (which helps you decide whether an effect is significant or not). It does not affect the calculations of the pvalues etc.
However, as long as you focus on pvalues and not confidence intervals you can compare the precise new threshold against the pvalues that you obtained; the coloring is just convenience.
Nevertheless, it may be a good suggestion to allow more decimals on the input value in the settings to allow researchers define the confidence intervals and coloring more precisely.
What you put in the significance field in SmartPLS 4, affects the confidence intervals and the coloring of the pvalues (which helps you decide whether an effect is significant or not). It does not affect the calculations of the pvalues etc.
However, as long as you focus on pvalues and not confidence intervals you can compare the precise new threshold against the pvalues that you obtained; the coloring is just convenience.
Nevertheless, it may be a good suggestion to allow more decimals on the input value in the settings to allow researchers define the confidence intervals and coloring more precisely.
Dr. JanMichael Becker, BI Norwegian Business School, SmartPLS Developer
Researchgate: https://www.researchgate.net/profile/Jan_Michael_Becker
GoogleScholar: http://scholar.google.de/citations?user ... AAAJ&hl=de
Researchgate: https://www.researchgate.net/profile/Jan_Michael_Becker
GoogleScholar: http://scholar.google.de/citations?user ... AAAJ&hl=de
Re: pvalue adjustment: 0.05 to 0.0169524 when executing the permutation test
Dear Dr. Becker
Thank you very much. I greatly appreciate your explanations; they have been incredibly beneficial.
Reagrds
Dasun
Thank you very much. I greatly appreciate your explanations; they have been incredibly beneficial.
Reagrds
Dasun