Holdout Sample calculation

Questions about the implementation and application of the PLS-SEM method, that are not related to the usage of the SmartPLS software.
Post Reply
janschreier
PLS Expert User
Posts: 116
Joined: Fri Sep 12, 2014 2:12 pm
Real name and title: Jan Schreier

Holdout Sample calculation

Post by janschreier »

Dear Forum-users,

since the available info on holdout samples with PLS-SEM - to my knowledge - is relatively limited, I tried to produce something for myself. So i created a macro with which you can easily take your models outer weights and the path cofficients and run this against the holdout sample data to obtain the R²-values for the holdout sample. As far as I understand the R²-values of the sample and the holdout sample should be fairly equal.

What I'm currently not 100% sure of is the data normalization process. Do you need to normalize all data (sample + holdout) together before doing anything in smartPLS or do you normalize both data sets separately?

Is this sufficient for a holdout sample test? Or do I need to do more?
If yes, this linked Excelsheet should do the trick with few clicks: http://bit.ly/1L2LnTY

Hope this helps somebody!
jan
User avatar
cringle
SmartPLS Developer
Posts: 818
Joined: Tue Sep 20, 2005 9:13 am
Real name and title: Prof. Dr. Christian M. Ringle
Location: Hamburg (Germany)
Contact:

Re: Holdout Sample calculation

Post by cringle »

Sounds like a good feature to implement into SmartPLS! Such example help us a lot to advance develoipments.

Best regards
Christian
janschreier
PLS Expert User
Posts: 116
Joined: Fri Sep 12, 2014 2:12 pm
Real name and title: Jan Schreier

Re: Holdout Sample calculation

Post by janschreier »

Good to hear, Prof. Ringle! I'm glad I can help :)

Would be pleased to get some feedback on the two methodical questions, too, though.

BR, Jan Schreier
Julen
PLS User
Posts: 17
Joined: Wed Apr 24, 2013 7:38 am
Real name and title:

Re: Holdout Sample calculation

Post by Julen »

Dear All,

You say that R2 values of the sample and the holdout sample should be fairly equal. Some breaf questions please:

1- Is there any reference / rule of thumb for assessing what is considered to be "fairly equal"?

2- If R2 values look too different, I guess the predictive validity would be problematic. How would we report it? for example noting that "the findings should be treated with caution since the predictive validity of the model is not large"? is that correct?

3- Is there a new feature in smartPLS last version related to predictive validity issues with holdout samples?

Thanks in advance!
nayabeyes
PLS Expert User
Posts: 74
Joined: Wed Nov 23, 2011 7:34 am
Real name and title:

Re: Holdout Sample calculation

Post by nayabeyes »

Hi Jan;
I am not sure if you have a chance to read Prof., Henesler file Assessing the Predictive Validity of PLS Path Models Using Holdout Samples?
I think that might be also a good help to confirm.
Best, Murad
janschreier wrote:Dear Forum-users,

since the available info on holdout samples with PLS-SEM - to my knowledge - is relatively limited, I tried to produce something for myself. So i created a macro with which you can easily take your models outer weights and the path cofficients and run this against the holdout sample data to obtain the R²-values for the holdout sample. As far as I understand the R²-values of the sample and the holdout sample should be fairly equal.

What I'm currently not 100% sure of is the data normalization process. Do you need to normalize all data (sample + holdout) together before doing anything in smartPLS or do you normalize both data sets separately?

Is this sufficient for a holdout sample test? Or do I need to do more?
If yes, this linked Excelsheet should do the trick with few clicks: http://bit.ly/1L2LnTY

Hope this helps somebody!
jan
MURAD ALI, Ph.D
janschreier
PLS Expert User
Posts: 116
Joined: Fri Sep 12, 2014 2:12 pm
Real name and title: Jan Schreier

Re: Holdout Sample calculation

Post by janschreier »

Dear Murad,

do you have a more exact source/reference for Prof. Henseler's paper?

Thanks for your help!
Jan
nayabeyes
PLS Expert User
Posts: 74
Joined: Wed Nov 23, 2011 7:34 am
Real name and title:

Re: Holdout Sample calculation

Post by nayabeyes »

Hi Jan,
Please send me your mail.
Best
janschreier wrote:Dear Murad,

do you have a more exact source/reference for Prof. Henseler's paper?

Thanks for your help!
Jan
MURAD ALI, Ph.D
Post Reply