p-value adjustment: 0.05 to 0.0169524 when executing the permutation test

[Capcap1668]

Dear everybody,

I am doing MICOM procedure of MGA (Multigroup analysis) for more than three groups. Due to a recommendation from a paper, the setting specifications: p value should be made adjustment (Bonferroni corrections): 0.05 to 0.0169524 when executing the permutation test.
However, in Permutation Multigroup Analysis of SmartPLS 4, the significance level automatically changes from 0.017 to 0.02.

Similarly, When I conduct Bootstrap multigroup analysis in Smartpls 4, the significance level automatically changes from 0.017 to 0.02.

How can I deal with this problem?

The link of paper recommending p value adjustment: https://www.sciencedirect.com/science/a ... 6322010049

I wish you a good working day ahead,

Best regard,

Hung

[dnirmala]

Dear everybody,

I am doing MICOM procedure of MGA (Multigroup analysis) for more than three groups. Due to a recommendation from a paper, the setting specifications: p value should be made adjustment (Bonferroni corrections): 0.05 to 0.0169524 when executing the permutation test.
However, in Permutation Multigroup Analysis of SmartPLS 4, the significance level automatically changes from 0.017 to 0.02.

Similarly, When I conduct Bootstrap multigroup analysis in Smartpls 4, the significance level automatically changes from 0.017 to 0.02.

How can I deal with this problem?

The link of paper recommending p value adjustment: https://www.sciencedirect.com/science/a ... 6322010049

I wish you a good working day ahead,

Best regard,

Hung

Hi everyone,
Thank you Hung for posting this question. I am also facing the same problem. Appreciate any help on how to handle these issues?

Regards
Dasun

[jmbecker]

To be honest, I don't understand what you mean with "the significance level automatically changes from 0.017 to 0.02."

If you run a (preferably permutation-based MGA), you will get a p-value for the difference in parameter between groups. You usually conclude that the difference is significant (whether significance this is a meaningful concept in its own is an entirely different question) when the p-value is below a specific threshold (usually 0.05, which represents your tolerable type I error rate). In the case of multiple pairwise group comparison you need to adjust for the inflation in error rate and lower the threshold (i.e., the 0.05). Thus, you compare your p-value against a lower threshold. The difference may then not be significant anymore.

Example:
The p-value for the difference in parameter A between group 1 and 2 is 0.03.
If you compare only the two groups you may conclude significance because 0.03 is smaller than 0.05 (i.e., it is significant on a 5% error level).
If you compare three groups you may not conclude significance because 0.03 is larger than 0.05/3=0.0167 (i.e., it is not significant on a 5% error level accounting for the inflation in error rate due to multiple group comparisons using Bonferoni).

The p-value for the difference in parameter B between group 1 and 2 is 0.006.
If you compare only the two groups you may conclude significance because 0.006 is smaller than 0.05 (i.e., it is significant on a 5% error level).
If you compare three groups you may also conclude significance because 0.006 is smaller than 0.05/3=0.0167 (i.e., it is significant on a 5% error level accounting for the inflation in error rate due to multiple group comparisons using Bonferoni).

[dnirmala]

To be honest, I don't understand what you mean with "the significance level automatically changes from 0.017 to 0.02."

If you run a (preferably permutation-based MGA), you will get a p-value for the difference in parameter between groups. You usually conclude that the difference is significant (whether significance this is a meaningful concept in its own is an entirely different question) when the p-value is below a specific threshold (usually 0.05, which represents your tolerable type I error rate). In the case of multiple pairwise group comparison you need to adjust for the inflation in error rate and lower the threshold (i.e., the 0.05). Thus, you compare your p-value against a lower threshold. The difference may then not be significant anymore.

Example:
The p-value for the difference in parameter A between group 1 and 2 is 0.03.
If you compare only the two groups you may conclude significance because 0.03 is smaller than 0.05 (i.e., it is significant on a 5% error level).
If you compare three groups you may not conclude significance because 0.03 is larger than 0.05/3=0.0167 (i.e., it is not significant on a 5% error level accounting for the inflation in error rate due to multiple group comparisons using Bonferoni).

The p-value for the difference in parameter B between group 1 and 2 is 0.006.
If you compare only the two groups you may conclude significance because 0.006 is smaller than 0.05 (i.e., it is significant on a 5% error level).
If you compare three groups you may also conclude significance because 0.006 is smaller than 0.05/3=0.0167 (i.e., it is significant on a 5% error level accounting for the inflation in error rate due to multiple group comparisons using Bonferoni).

Dear Dr. Becker
Thank you for the explanation.
I was under the impression that we need to run the 'Permutation multigroup' procedure by adding the Šidák p value adjustment (0.0167) as the significance level to the procedure. When we did that software automatically round up the value to 0.2. (Please see below screenshot). Cheah et al. (2023) also mentioned about this.

In this study, the p-value adjustment based on the Sid ˇ ak ´ procedure was used to assess the compositional invariance, the composites’ equality of mean values, and variances of the composites. If SmartPLS 3 is used, the adjustment of the permutation test will be in three decimal point that is 0.017. In contrast, if SmartPLS4 is used, the adjustment of the permutation test will be in two decimal point that is 0.02.

(Cheah et al., 2023, p. 10)

What I understood from your answer is that we need to run the MGA with 0.05 as significance level but assess the result using Šidák p value adjustment.

Thank you!

Regards
Dasun

Screenshot 2023-10-14 092024.png (65.49 KiB) Viewed 1841 times

Reference
Cheah, J.-H., Amaro, S., & Roldán, J. L. (2023). Multigroup analysis of more than two groups in PLS-SEM: A review, illustration, and recommendations. Journal of business research, 156, 113539. https://doi.org/https://doi.org/10.1016 ... 022.113539

[jmbecker]

First, whether you use Sidak or Bonferroni is your choice that you need to justify. I have not seen clear evidence that the one is always superior over the other. I was using Bonferoni, because it simplified the example as it is much simpler to calculate.

What you put in the significance field in SmartPLS 4, affects the confidence intervals and the coloring of the p-values (which helps you decide whether an effect is significant or not). It does not affect the calculations of the p-values etc.
However, as long as you focus on p-values and not confidence intervals you can compare the precise new threshold against the p-values that you obtained; the coloring is just convenience.

Nevertheless, it may be a good suggestion to allow more decimals on the input value in the settings to allow researchers define the confidence intervals and coloring more precisely.

[dnirmala]

Dear Dr. Becker

Thank you very much. I greatly appreciate your explanations; they have been incredibly beneficial.

Reagrds
Dasun