towardsdatascience.com Open in urlscan Pro
162.159.152.4 Public Scan

Back to summary

Submitted URL:
https://towardsdatascience.com/kernel-pca-vs-pca-vs-ica-in-tensorflow-sklearn-60e17eb15a64
Effective URL:
https://towardsdatascience.com/kernel-pca-vs-pca-vs-ica-in-tensorflow-sklearn-60e17eb15a64?gi=ecb766a4c265
Submission Tags: falconsandbox
Submission: On September 18 via api (September 18th 2022, 2:47:27 am UTC) from US — Scanned from DE

Form analysis
0 forms found in the DOM

Text Content

Open in app

Get started

Home
Notifications
Lists
Stories

--------------------------------------------------------------------------------

Write

Published in

Towards Data Science

Jae Duk Seo
Follow

Sep 10, 2018

·
6 min read
·

Listen

Save

KERNEL PCA VS PCA VS ICA IN TENSORFLOW/SKLEARN

GIF from this website

Principle Component Analysis performs a linear transformation on a given data,
however, many real-world data are not linearly separable. So can we take
advantage of higher dimensions, while not increasing the needed computation
power so much?

> Please note that this post is for my future self to look back and review the
> materials on this post. (and self study)

Lecture: Kernel PCA

PPT from this website

Form the PPT above, I will make some short notes that were helpful to me.

Vapnik–Chervonenkis theory, tells us that if we project our data into a higher
dimensional space, it provides us with better classification power. (Example
seen left.) This might be similar to what a neural network us doing overall, as
the depth increase more abstract features are extracted and have better features
to perform classification.

kernel trick, a method to project original data into higher dimension without
sacrificing too much computational time. (Non-linear feature mapping). And the
matrix form to normalize the feature space.

Example of effective use of KPCA, seen above.

Different Use Cases of KPCA

Paper from this website

In the first paper, the authors uses KPCA as a preprocessing step as a mean of
feature transformation, and paired up with Least Squares Support Vector Machine
to perform classification on DNA micro-arrays. (Micro-array data have high
dimensions and thus it is a good idea to perform dimensionality reduction
techniques before performing classification.) In the second paper KPCA was use
to extract features from functional Magnetic Resonance Image(fMRI) to perform
automatic diagnostic for Attention-Deficit Hyperactivity Disorder(ADHD).

KPCA (RBF) Layer In Tensorflow

A simple feed forward operation can be implemented like above, and at the time
of writing this article, I won’t work my way into implementing back propagation
respect to the input data.

KPCA vs PCA vs ICA

Lets start simple, we have a 2D data points that is linearly inseparable and now
to verify that our implementation is working lets project our data into two
dimensional space, using each KPCA, PCA and ICA.

Left Image → Projection using KPCA
Middle Image → Projection using PCA
Right Image → Projection using ICA

From the above example we can see that our implementation is working correctly
and our data is now linearly separable. But to make things more interesting lets
see how these methods will do on histopathological images. I am using the data
set from Histopathology data of bone marrow biopsies (HistBMP).

As seen above, each image is 28*28 gray scale image and we are going to find the
eigen images by compressing 1000 image into 100.

Left Image → Projection using KPCA
Middle Image → Projection using PCA
Right Image → Projection using ICA

In general we can see that PCA tries to capture global changes, ICA tries to
capture local changes. But KPCA seems to first capture global changes but as we
get to the lower part of eigen images, we can see that it is capturing local
changes.

Code

For Google Colab, you would need a google account to view the codes, also you
can’t run read only scripts in Google Colab so make a copy on your play ground.
Finally, I will never ask for permission to access your files on Google Drive,
just FYI. Happy Coding!

To access the code for this post please click here.

Final Words

Please note that for distance matrix I borrowed the non-loop form from this
website, and overall implementation was borrowed from ‘Kernel tricks and
nonlinear dimensionality reduction via RBF kernel PCA’ by Sebastian Raschka.

I always wondered how we can plot how much variance we can keep for each
individual eigenvalues, and this was a good post that explained the know how.

Image from this website

Also this was an interesting video I found.

video from this website

Finally, it was interesting to know that PCA / KPCA suffers from variance
inflation and lack of generalizability, the paper below proposes a solution to
the problem.

Paper from this website

If any errors are found, please email me at jae.duk.seo@gmail.com, if you wish
to see the list of all of my writing please view my website here.

Meanwhile follow me on my twitter here, and visit my website, or my Youtube
channel for more content. I also implemented Wide Residual Networks, please
click here to view the blog post.

Reference

1. Principal Component Analysis. (2015). Dr. Sebastian Raschka. Retrieved 7
September 2018, from
https://sebastianraschka.com/Articles/2015_pca_in_3_steps.html
2. About Feature Scaling and Normalization. (2014). Dr. Sebastian Raschka.
Retrieved 7 September 2018, from
https://sebastianraschka.com/Articles/2014_about_feature_scaling.html
3. About Feature Scaling and Normalization. (2014). Dr. Sebastian Raschka.
Retrieved 7 September 2018, from
https://sebastianraschka.com/Articles/2014_about_feature_scaling.html
4. Implementing a Principal Component Analysis (PCA). (2014). Dr. Sebastian
Raschka. Retrieved 7 September 2018, from
https://sebastianraschka.com/Articles/2014_pca_step_by_step.html
5. tf.ones | TensorFlow. (2018). TensorFlow. Retrieved 10 September 2018, from
https://www.tensorflow.org/api_docs/python/tf/ones
6. Distance Matrix Vectorization Trick — Manifold Blog — Medium. (2016).
Medium. Retrieved 10 September 2018, from
https://medium.com/dataholiks-distillery/l2-distance-matrix-vectorization-trick-26aa3247ac6c
7. matplotlib, P. (2018). Plot two histograms at the same time with
matplotlib. Stack Overflow. Retrieved 10 September 2018, from
https://stackoverflow.com/questions/6871201/plot-two-histograms-at-the-same-time-with-matplotlib
8. tf.self_adjoint_eig | TensorFlow. (2018). TensorFlow. Retrieved 10
September 2018, from
https://www.tensorflow.org/api_docs/python/tf/self_adjoint_eig
9. Kernel tricks and nonlinear dimensionality reduction via RBF kernel PCA.
(2014). Dr. Sebastian Raschka. Retrieved 10 September 2018, from
https://sebastianraschka.com/Articles/2014_kernel_pca.html
10. Vapnik–Chervonenkis theory. (2018). En.wikipedia.org. Retrieved 10
September 2018, from
https://en.wikipedia.org/wiki/Vapnik%E2%80%93Chervonenkis_theory
11. Sidhu, G., Asgarian, N., Greiner, R., & Brown, M. (2012). Kernel Principal
Component Analysis for dimensionality reduction in fMRI-based diagnosis of
ADHD. Frontiers In Systems Neuroscience, 6. doi:10.3389/fnsys.2012.00074
12. Thomas, M., Brabanter, K., & Moor, B. (2014). New bandwidth selection
criterion for Kernel PCA: Approach to dimensionality reduction and
classification problems. BMC Bioinformatics, 15(1), 137.
doi:10.1186/1471–2105–15–137
13. Abrahamsen, T., & Hansen, L. (2011). A Cure for Variance Inflation in High
Dimensional Kernel Principal Component Analysis. Journal Of Machine
Learning Research, 12(Jun), 2027–2044. Retrieved from
http://jmlr.csail.mit.edu/papers/v12/abrahamsen11a.html
14. Tomczak, J. (2018). Histopathology data of bone marrow biopsies (HistBMP).
Zenodo. Retrieved 10 September 2018, from
https://zenodo.org/record/1205024#.W5bcCOhKiUm

241

BY TOWARDS DATA SCIENCE

Every Thursday, the Variable delivers the very best of Towards Data Science:
from hands-on tutorials and cutting-edge research to original features you don't
want to miss. Take a look.

By signing up, you will create a Medium account if you don’t already have one.
Review our Privacy Policy for more information about our privacy practices.

Get this newsletter

towardsdatascience.com Open in urlscan Pro 162.159.152.4 Public Scan

Form analysis 0 forms found in the DOM

Text Content

towardsdatascience.com Open in urlscan Pro
162.159.152.4 Public Scan

Form analysis
0 forms found in the DOM