Holdout Sample calculation

Frequently asked questions about PLS path modeling.
Post Reply
janschreier
PLS Expert User
Posts: 116
Joined: Fri Sep 12, 2014 2:12 pm
Real name and title: Jan Schreier

Holdout Sample calculation

Post by janschreier » Thu Jan 22, 2015 12:47 pm

Dear Forum-users,

since the available info on holdout samples with PLS-SEM - to my knowledge - is relatively limited, I tried to produce something for myself. So i created a macro with which you can easily take your models outer weights and the path cofficients and run this against the holdout sample data to obtain the R²-values for the holdout sample. As far as I understand the R²-values of the sample and the holdout sample should be fairly equal.

What I'm currently not 100% sure of is the data normalization process. Do you need to normalize all data (sample + holdout) together before doing anything in smartPLS or do you normalize both data sets separately?

Is this sufficient for a holdout sample test? Or do I need to do more?
If yes, this linked Excelsheet should do the trick with few clicks: http://bit.ly/1L2LnTY

Hope this helps somebody!
jan

User avatar
cringle
SmartPLS Developer
Posts: 812
Joined: Tue Sep 20, 2005 9:13 am
Real name and title: Prof. Dr. Christian M. Ringle
Location: Hamburg (Germany)
Contact:

Re: Holdout Sample calculation

Post by cringle » Fri Jan 23, 2015 8:54 am

Sounds like a good feature to implement into SmartPLS! Such example help us a lot to advance develoipments.

Best regards
Christian

janschreier
PLS Expert User
Posts: 116
Joined: Fri Sep 12, 2014 2:12 pm
Real name and title: Jan Schreier

Re: Holdout Sample calculation

Post by janschreier » Fri Jan 23, 2015 9:58 am

Good to hear, Prof. Ringle! I'm glad I can help :)

Would be pleased to get some feedback on the two methodical questions, too, though.

BR, Jan Schreier

Julen
PLS User
Posts: 17
Joined: Wed Apr 24, 2013 7:38 am
Real name and title:

Re: Holdout Sample calculation

Post by Julen » Mon Nov 09, 2015 10:17 am

Dear All,

You say that R2 values of the sample and the holdout sample should be fairly equal. Some breaf questions please:

1- Is there any reference / rule of thumb for assessing what is considered to be "fairly equal"?

2- If R2 values look too different, I guess the predictive validity would be problematic. How would we report it? for example noting that "the findings should be treated with caution since the predictive validity of the model is not large"? is that correct?

3- Is there a new feature in smartPLS last version related to predictive validity issues with holdout samples?

Thanks in advance!

nayabeyes
PLS Expert User
Posts: 74
Joined: Wed Nov 23, 2011 7:34 am
Real name and title:

Re: Holdout Sample calculation

Post by nayabeyes » Thu Jan 21, 2016 10:12 am

Hi Jan;
I am not sure if you have a chance to read Prof., Henesler file Assessing the Predictive Validity of PLS Path Models Using Holdout Samples?
I think that might be also a good help to confirm.
Best, Murad
janschreier wrote:Dear Forum-users,

since the available info on holdout samples with PLS-SEM - to my knowledge - is relatively limited, I tried to produce something for myself. So i created a macro with which you can easily take your models outer weights and the path cofficients and run this against the holdout sample data to obtain the R²-values for the holdout sample. As far as I understand the R²-values of the sample and the holdout sample should be fairly equal.

What I'm currently not 100% sure of is the data normalization process. Do you need to normalize all data (sample + holdout) together before doing anything in smartPLS or do you normalize both data sets separately?

Is this sufficient for a holdout sample test? Or do I need to do more?
If yes, this linked Excelsheet should do the trick with few clicks: http://bit.ly/1L2LnTY

Hope this helps somebody!
jan
MURAD ALI, Ph.D

janschreier
PLS Expert User
Posts: 116
Joined: Fri Sep 12, 2014 2:12 pm
Real name and title: Jan Schreier

Re: Holdout Sample calculation

Post by janschreier » Thu Jan 21, 2016 10:17 am

Dear Murad,

do you have a more exact source/reference for Prof. Henseler's paper?

Thanks for your help!
Jan

nayabeyes
PLS Expert User
Posts: 74
Joined: Wed Nov 23, 2011 7:34 am
Real name and title:

Re: Holdout Sample calculation

Post by nayabeyes » Thu Jan 21, 2016 10:19 am

Hi Jan,
Please send me your mail.
Best
janschreier wrote:Dear Murad,

do you have a more exact source/reference for Prof. Henseler's paper?

Thanks for your help!
Jan
MURAD ALI, Ph.D

Post Reply