Dear Forum-users,
since the available info on holdout samples with PLS-SEM - to my knowledge - is relatively limited, I tried to produce something for myself. So i created a macro with which you can easily take your models outer weights and the path cofficients and run this against the holdout sample data to obtain the R²-values for the holdout sample. As far as I understand the R²-values of the sample and the holdout sample should be fairly equal.
What I'm currently not 100% sure of is the data normalization process. Do you need to normalize all data (sample + holdout) together before doing anything in smartPLS or do you normalize both data sets separately?
Is this sufficient for a holdout sample test? Or do I need to do more?
If yes, this linked Excelsheet should do the trick with few clicks: http://bit.ly/1L2LnTY
Hope this helps somebody!
jan
Holdout Sample calculation
-
- PLS Expert User
- Posts: 116
- Joined: Fri Sep 12, 2014 2:12 pm
- Real name and title: Jan Schreier
- cringle
- SmartPLS Developer
- Posts: 818
- Joined: Tue Sep 20, 2005 9:13 am
- Real name and title: Prof. Dr. Christian M. Ringle
- Location: Hamburg (Germany)
- Contact:
Re: Holdout Sample calculation
Sounds like a good feature to implement into SmartPLS! Such example help us a lot to advance develoipments.
Best regards
Christian
Best regards
Christian
Prof. Dr. Christian M. Ringle, Hamburg University of Technology (TUHH), SmartPLS
- Literature on PLS-SEM: https://www.smartpls.com/documentation
- Google Scholar: https://scholar.google.de/citations?use ... AAAJ&hl=de
- Literature on PLS-SEM: https://www.smartpls.com/documentation
- Google Scholar: https://scholar.google.de/citations?use ... AAAJ&hl=de
-
- PLS Expert User
- Posts: 116
- Joined: Fri Sep 12, 2014 2:12 pm
- Real name and title: Jan Schreier
Re: Holdout Sample calculation
Good to hear, Prof. Ringle! I'm glad I can help :)
Would be pleased to get some feedback on the two methodical questions, too, though.
BR, Jan Schreier
Would be pleased to get some feedback on the two methodical questions, too, though.
BR, Jan Schreier
Re: Holdout Sample calculation
Dear All,
You say that R2 values of the sample and the holdout sample should be fairly equal. Some breaf questions please:
1- Is there any reference / rule of thumb for assessing what is considered to be "fairly equal"?
2- If R2 values look too different, I guess the predictive validity would be problematic. How would we report it? for example noting that "the findings should be treated with caution since the predictive validity of the model is not large"? is that correct?
3- Is there a new feature in smartPLS last version related to predictive validity issues with holdout samples?
Thanks in advance!
You say that R2 values of the sample and the holdout sample should be fairly equal. Some breaf questions please:
1- Is there any reference / rule of thumb for assessing what is considered to be "fairly equal"?
2- If R2 values look too different, I guess the predictive validity would be problematic. How would we report it? for example noting that "the findings should be treated with caution since the predictive validity of the model is not large"? is that correct?
3- Is there a new feature in smartPLS last version related to predictive validity issues with holdout samples?
Thanks in advance!
Re: Holdout Sample calculation
Hi Jan;
I am not sure if you have a chance to read Prof., Henesler file Assessing the Predictive Validity of PLS Path Models Using Holdout Samples?
I think that might be also a good help to confirm.
Best, Murad
I am not sure if you have a chance to read Prof., Henesler file Assessing the Predictive Validity of PLS Path Models Using Holdout Samples?
I think that might be also a good help to confirm.
Best, Murad
janschreier wrote:Dear Forum-users,
since the available info on holdout samples with PLS-SEM - to my knowledge - is relatively limited, I tried to produce something for myself. So i created a macro with which you can easily take your models outer weights and the path cofficients and run this against the holdout sample data to obtain the R²-values for the holdout sample. As far as I understand the R²-values of the sample and the holdout sample should be fairly equal.
What I'm currently not 100% sure of is the data normalization process. Do you need to normalize all data (sample + holdout) together before doing anything in smartPLS or do you normalize both data sets separately?
Is this sufficient for a holdout sample test? Or do I need to do more?
If yes, this linked Excelsheet should do the trick with few clicks: http://bit.ly/1L2LnTY
Hope this helps somebody!
jan
MURAD ALI, Ph.D
-
- PLS Expert User
- Posts: 116
- Joined: Fri Sep 12, 2014 2:12 pm
- Real name and title: Jan Schreier
Re: Holdout Sample calculation
Dear Murad,
do you have a more exact source/reference for Prof. Henseler's paper?
Thanks for your help!
Jan
do you have a more exact source/reference for Prof. Henseler's paper?
Thanks for your help!
Jan
Re: Holdout Sample calculation
Hi Jan,
Please send me your mail.
Best
Please send me your mail.
Best
janschreier wrote:Dear Murad,
do you have a more exact source/reference for Prof. Henseler's paper?
Thanks for your help!
Jan
MURAD ALI, Ph.D