How to choose the number of features?

[GuannanZhao]

I just find this at the tutorial page of BrainIAK for SRM:

Ideally, the number of features should be chosen using cross-validation. We will follow the lead of Chen et al. (2015) for now and use 50 features. Typically, 20 iterations should be enough for SRM to converge.

How could I choose the features by cross-validation?
How could I know which number of features is the best?

Sorry for asking such a basic question.

Thanks,
Zhao

[cameronphchen]

Treat the number of features as hyperparameter and select the one with the highest performance in cross-validation.

…

On Tue, Sep 17, 2019, 03:52 GuannanZhao ***@***.***> wrote: I just find this at the tutorial page of BrainIAK for SRM: Ideally, the number of features should be chosen using cross-validation. We will follow the lead of Chen et al. (2015) for now and use 50 features. Typically, 20 iterations should be enough for SRM to converge. How could I choose the features by cross-validation? How could I know which number of features is the best? Sorry for asking such a basic question. Thanks, Zhao — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#1?email_source=notifications&email_token=AAVISEXSDGSPLYIUIHCQOUDQKCZGTA5CNFSM4IXODPTKYY3PNVWWK3TUL52HS4DFUVEXG43VMWVGG33NNVSW45C7NFSM4HL2HEOQ>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAVISEW45V5PMK5X3Z6W6J3QKCZGTANCNFSM4IXODPTA> .

[GuannanZhao]

Thank you for the reply! @cameronphchen

I noticed that but still not quite sure about the definition of performance when testing with ISC.
As written in the tutorial/11-SRM, the performance is "defined as the difference between raw and shared ISC".

But the given example of performance is a t-test between raw and shared ISC on test data. I tried k=10, 50, 100, 500, 1000(voxel number is 1000) and only to find that the average correlation is more likely to be small. So whether the ''difference" means "difference of average correlation between raw and shared ISC" or just the "p-value of t-test between raw and share ISC"?
Since the p-value for different k between raw and shared ISC are all the same(p = 0.0), it seems that in this tutorial all k values are Ok to train a SRM model.

Did I wrongly understand the tutorial or miss something? Looking forward to your reply.

Thanks again!