All,
not sure really, if this is a bug or a feature...
I've noticed that SmartPLS and LVPLS v1.8 give very different estimates for structural paths and indicator weights/loadings for models that are specified in an identical way and estimated using the same raw data. In some cases, the differences are small, in others, the difference is between statistical significance and a value near 0 for structural paths.
As far as I understand, both programs should yield the same estimation results provided the data and model specification are the same, or shouldn't they?
Can somebody confirm this observation?
Maybe there are small but important differences in the estimation parameters that I have not noticed (despite trying my best to rule this out). However, I would very much like to understand whether there are systematic differences between the estimation procedures used in SmartPLS and LVPLS and identify the source of this discrepancy.
Any comments from the development team on this? Have you run any tests to compare SmartPLS output with PLSGraph or LVPLS?
Regards,
Stefan
Estimation differences SmartPLS vs. LVPLS
-
stefanbehrens
- PLS Expert User
- Posts: 54
- Joined: Wed Oct 19, 2005 5:53 pm
- Real name and title:
-
stefanbehrens
- PLS Expert User
- Posts: 54
- Joined: Wed Oct 19, 2005 5:53 pm
- Real name and title:
-
cringle
- SmartPLS Developer
- Posts: 818
- Joined: Tue Sep 20, 2005 9:13 am
- Real name and title: Prof. Dr. Christian M. Ringle
- Location: Hamburg (Germany)
- Contact:
Hi,
Dirk Temme and Henning Kreis (see viewtopic.php?t=20 and viewtopic.php?t=19) already checked this: there are no differences in the numbers but there might be some sign changes due to different initilizations of the algorith.
Dirk Temme and Henning Kreis (see viewtopic.php?t=20 and viewtopic.php?t=19) already checked this: there are no differences in the numbers but there might be some sign changes due to different initilizations of the algorith.
Prof. Dr. Christian M. Ringle, Hamburg University of Technology (TUHH), SmartPLS
- Literature on PLS-SEM: https://www.smartpls.com/documentation
- Google Scholar: https://scholar.google.de/citations?use ... AAAJ&hl=de
- Literature on PLS-SEM: https://www.smartpls.com/documentation
- Google Scholar: https://scholar.google.de/citations?use ... AAAJ&hl=de
-
stefanbehrens
- PLS Expert User
- Posts: 54
- Joined: Wed Oct 19, 2005 5:53 pm
- Real name and title:
Alex,
Christian,
I think I figured out what caused the differences in path estimates. It had to do with the treatment of missing values. Somehow, I had calclulated my models without the explicit missing value treatment (I thought the default was "replace with mean" as in SmartPLS 1.0) and consequently, the "-1"s in the survey data were included in the mean/stdev calculations for the variable standardization. Obviously this yielded quite different results from the LVPLS calculations. Now that I have explicitely turned on the missing value pre-processing option in SmartPLS 2.0, the results are consistent with LVPLS up to the second decimal.
The remaining (small) differences I was able to trace to small variances in the MV-scores after standardization. My guess is that LVPLS calculates with less precision in the standardization procedure...
In a nutshell, this means that cross-program comparisons can only be meaningful if carried out on pre-standardized raw data.
Cheers,
Stefan
PS: This also means that I have to recalculate all my models as the previous results are all completely invalid :-((
Christian,
I think I figured out what caused the differences in path estimates. It had to do with the treatment of missing values. Somehow, I had calclulated my models without the explicit missing value treatment (I thought the default was "replace with mean" as in SmartPLS 1.0) and consequently, the "-1"s in the survey data were included in the mean/stdev calculations for the variable standardization. Obviously this yielded quite different results from the LVPLS calculations. Now that I have explicitely turned on the missing value pre-processing option in SmartPLS 2.0, the results are consistent with LVPLS up to the second decimal.
The remaining (small) differences I was able to trace to small variances in the MV-scores after standardization. My guess is that LVPLS calculates with less precision in the standardization procedure...
In a nutshell, this means that cross-program comparisons can only be meaningful if carried out on pre-standardized raw data.
Cheers,
Stefan
PS: This also means that I have to recalculate all my models as the previous results are all completely invalid :-((
-
cringle
- SmartPLS Developer
- Posts: 818
- Joined: Tue Sep 20, 2005 9:13 am
- Real name and title: Prof. Dr. Christian M. Ringle
- Location: Hamburg (Germany)
- Contact:
Stefan,
in a nutshell:
you are not reporting a bug!
You are just describing your fault not use the missing value option while there are missing values in your data basis that are indicated by a specific number! Such a mistake is also possible in LVPLS/PLSgraph – you need to indicate the missing values. We are aware of this problem – that’s why we included a step by step data import dialog that explicitly asks you if there are missing values in your data and the used value is to indicate them.
My advice: Use a certain treatment of missing values before you import your data into SmartPLS! Then you make sure to avoid this problem.
Best
Christian
in a nutshell:
you are not reporting a bug!
You are just describing your fault not use the missing value option while there are missing values in your data basis that are indicated by a specific number! Such a mistake is also possible in LVPLS/PLSgraph – you need to indicate the missing values. We are aware of this problem – that’s why we included a step by step data import dialog that explicitly asks you if there are missing values in your data and the used value is to indicate them.
My advice: Use a certain treatment of missing values before you import your data into SmartPLS! Then you make sure to avoid this problem.
That's correct.My guess is that LVPLS calculates with less precision in the standardization procedure...
That's not correct - a missing value treatment is required in both programs leading to similar results.... cross-program comparisons can only be meaningful if carried out on pre-standardized raw data
Best
Christian
Prof. Dr. Christian M. Ringle, Hamburg University of Technology (TUHH), SmartPLS
- Literature on PLS-SEM: https://www.smartpls.com/documentation
- Google Scholar: https://scholar.google.de/citations?use ... AAAJ&hl=de
- Literature on PLS-SEM: https://www.smartpls.com/documentation
- Google Scholar: https://scholar.google.de/citations?use ... AAAJ&hl=de
-
stefanbehrens
- PLS Expert User
- Posts: 54
- Joined: Wed Oct 19, 2005 5:53 pm
- Real name and title:
Christian,
you are right that what I reported was simply "human error" (i.e. my stupidity) and not a bug. However, I would guess that I'm not the only one out there stupid enough to run into this (maybe others are simply not yet aware of it). What was confusing me (and maybe this is a suggestion for future releases of the program) was the way you have to configure the missing data treatment in the software. Let me explain:
- Before importing my data, I replaced missing values with "-1.0" (in SPSS)
- When I imported my data into SmartPLS, I checked the "indicator data contains missing values" box and used "-1.0" as "missing value".
- Then I went to "Settings"/ "Missing Value Processors" and selected "Mean Replacement"
So far, so intuitive. That's when I thought I had done everything necessary to tell the program how I wanted to treat my missing values and thus started to calculate my models. However, I was missing a final step (as I found out trying to compare the results between SmartPLS and LVPLS):
- I also had to go to "Settings"/ "Data Preprocessing", click "New..." and select "Missing Value Treatment"
It is only that last step that threw me off. The remaining procedure you guys devised was very straightforward. Maybe you could turn that last step on by default in future releases or hide the "Missing Values Processors"-dialog if no preprocessor is active.
All the best,
Stefan
PS: Maybe this thread can be moved to the FAQ section since it is not really concerning a bug as we both agree.
you are right that what I reported was simply "human error" (i.e. my stupidity) and not a bug. However, I would guess that I'm not the only one out there stupid enough to run into this (maybe others are simply not yet aware of it). What was confusing me (and maybe this is a suggestion for future releases of the program) was the way you have to configure the missing data treatment in the software. Let me explain:
- Before importing my data, I replaced missing values with "-1.0" (in SPSS)
- When I imported my data into SmartPLS, I checked the "indicator data contains missing values" box and used "-1.0" as "missing value".
- Then I went to "Settings"/ "Missing Value Processors" and selected "Mean Replacement"
So far, so intuitive. That's when I thought I had done everything necessary to tell the program how I wanted to treat my missing values and thus started to calculate my models. However, I was missing a final step (as I found out trying to compare the results between SmartPLS and LVPLS):
- I also had to go to "Settings"/ "Data Preprocessing", click "New..." and select "Missing Value Treatment"
It is only that last step that threw me off. The remaining procedure you guys devised was very straightforward. Maybe you could turn that last step on by default in future releases or hide the "Missing Values Processors"-dialog if no preprocessor is active.
All the best,
Stefan
PS: Maybe this thread can be moved to the FAQ section since it is not really concerning a bug as we both agree.
-
swende
- SmartPLS Developer
- Posts: 111
- Joined: Mon Sep 26, 2005 10:34 am
- Real name and title: Dipl. Wi-Inf. Sven Wende
- Location: Hamburg (Germany)
You are right. That might put off many other people, too. We will reconsider this and find a more intuitive solution until the next release.I also had to go to "Settings"/ "Data Preprocessing", click "New..." and select "Missing Value Treatment"
It is only that last step that threw me off.
Sven Wende, CEO SmartPLS GmbH