Uncategorized

Kaggle: Food 101. Retrained VGG16 model: 78% accuracy on 101,000 images across 101 classes.

Summary

101 different food categories across 101,000 images.
Transfer learning on a VGG16 model can be downloaded here.
Kaggle jupyter notebook is here.

Background

This is the Food 101 dataset, also available from https://www.vision.ee.ethz.ch/datasets_extra/food-101/

“Food-101 – Mining Discriminative Components with Random Forests” by Lukas Bossard, Matthieu Guillaumin and Luc Van Gool cir 2014 had an average accuracy of 50.76%.

Acknowledgements

  • The Food-101 data set consists of images from Foodspotting [1] which are not property of the Federal Institute of Technology Zurich (ETHZ). Any use beyond scientific fair use must be negotiated with the respective picture owners according to the Foodspotting terms of use [2].

[1] http://www.foodspotting.com/ [2] http://www.foodspotting.com/terms/

Accomplishment:

Was able to achieve 78% accuracy on test set (train/test split) using an GTX 1080ti over a period of 2 days. Training images were heavily augmented. Started off with a non-trainable VGG16 image-net pretrained model. Investigated and analyzed the features within the bottleneck using TensorBoard and other tools; determined sufficient feature transfer to continue; expanded out to 3 dense layers.

Conclusion:

VGG16 is quite amazing, obviously VGG19 does better when dealing with more classes and I didn’t expect this model to do so good with this little work on such a high number of classes.

Standard
Uncategorized

Kaggle: NOAA Fisheries Steller Sea Lion Population Count

Steller sea lions in the western Aleutian Islands have declined 94 percent in the last 30 years. The endangered western population, found in the North Pacific, are the focus of conservation efforts which require annual population counts. Specially trained scientists at NOAA Fisheries Alaska Fisheries Science Center conduct these surveys using airplanes and unoccupied aircraft systems to collect aerial images. Having accurate population estimates enables us to better understand factors that may be contributing to lack of recovery of Stellers in this area.

Currently, it takes biologists up to four months to count sea lions from the thousands of images NOAA Fisheries collects each year. Once individual counts are conducted, the tallies must be reconciled to confirm their reliability. The results of these counts are time-sensitive.

In this competition, Kagglers are invited to develop algorithms which accurately count the number of sea lions in aerial photographs. Automating the annual population count will free up critical resources allowing NOAA Fisheries to focus on ensuring we hear the sea lion’s roar for many years to come. Plus, advancements in computer vision applied to aerial population counts may also greatly benefit other endangered species.

These attachments represent roughly two weeks of work, getting roughly 90% accuracy. CLICK HERE v1 CLICK HERE v6

Standard
Uncategorized

Resume

Data Scientist (M.S.), Biochemist (B.S.), and C# .NET/Java/Python Software Engineer (MCAD) with a proven success record. Experience with big data using spark/hadoop, AI, Machine Learning, NLP. Excellent at gathering requirements from business units and scientists, and creating ad-hoc reports in SQL in an Agile environment.

 

Click HERE to see my resume

Standard
Uncategorized

Useful spark console logging

cd /etc/spark/conf
sudo nano log4j.properties
‘log4j.rootCategory=INFO,console’ -> ‘log4j.rootCategory=WARN,console’

get’s rid of the useless INFO messages!

f name == “main“:
sc = SparkContext()
log4jLogger = sc._jvm.org.apache.log4j
log = log4jLogger.LogManager.getLogger(name)
log.warn(“Custom Logging message that you can see on the console!”)

:boom::boom::boom::boom::boom::boom::boom::boom::boom::boom::boom::boom::boom::boom::boom::boom:

Standard
Uncategorized

Building TensorFlow 1.3 from Source on mac osx. Sierra. macbook pro. i7. with SSE and AVX

Stock TensorFlow for mac OSX (no nvida GPU) will give the warnings that it was compiled without SSE and AVX instructions.

The TensorFlow library wasn't compiled to use SSE4.1 instructions, but these are available on your machine and could speed up CPU computations.
The TensorFlow library wasn't compiled to use SSE4.2 instructions, but these are available on your machine and could speedup CPU computations.
The TensorFlow library wasn't compiled to use AVX instructions, but these are available on your machine and could speedup CPU computations.

/*

TF built to use these instructions will give a ~30% performance boost.

37210.gif

 

It’s weird that google does not provide a tensorflow version compiled for SSE and AVX, but whatever. Listed below is a script that allows for SSE and AVX instructions.

 

 

do each of these steps independently

will take around 1hr to complete all the steps regardless of machine type

*/

one time install

install anaconda3 pkg # manually download this and install the package
conda update conda
conda create -n dl python=3.6 anaconda
source activate dl

cd /
brew install bazel
pip install six numpy wheel
pip install –upgrade https://storage.googleapis.com/tensorflow/mac/cpu/protobuf-3.1.0-cp35-none-macosx_10_11_x86_64.whl

sudo -i
cd /
rm -rf tensorflow # if rerunning the script
cd /
git clone https://github.com/tensorflow/tensorflow

Step 1

cd /tensorflow
git checkout r1.3 -f
cd /
chmod -R 777 tensorflow
cd /tensorflow
./configure # accept all default settings

Step 2

// https://stackoverflow.com/questions/41293077/how-to-compile-tensorflow-with-sse4-2-and-avx-instructions
bazel build –config=opt –copt=-mavx –copt=-mavx2 –copt=-mfma //tensorflow/tools/pip_package:build_pip_package

Step 3

bazel-bin/tensorflow/tools/pip_package/build_pip_package /tmp/tensorflow_pkg
pip install /tmp/tensorflow_pkg/tensorflow-1.3.0rc1-cp36-cp36m-macosx_10_7_x86_64.whl

Step 4

cd ~
ipython

Step 5

import tensorflow as tf
hello = tf.constant(‘Hello, TensorFlow!’)
sess = tf.Session()
print(sess.run(hello))
print(tf.version)

Step 6

pip uninstall /tmp/tensorflow_pkg/tensorflow-1.3.0rc1-cp36-cp36m-macosx_10_7_x86_64.whl

Standard
Uncategorized

Building TensorFlow 1.2 from Source on mac osx. Sierra. macbook pro. i7. with SSE and AVX

/*

do each of these steps independently

will take around 1hr to complete all the steps regardless of machine type

*/

one time install

install anaconda3 pkg
conda update conda
conda create -n dl python=3.6 anaconda
source activate dl

cd /
brew install bazel
pip install six numpy wheel
pip install –upgrade https://storage.googleapis.com/tensorflow/mac/cpu/protobuf-3.1.0-cp35-none-macosx_10_11_x86_64.whl

sudo -i

cd /
rm -rf tensorflow # if rerunning the script

cd /
git clone https://github.com/tensorflow/tensorflow # is it really necessary to do this everytime?!

Step 1

cd /tensorflow
git checkout r1.2
cd /
chmod -R 777 tensorflow
cd /tensorflow
./configure # accept all default settings

Step 2

https://stackoverflow.com/questions/41293077/how-to-compile-tensorflow-with-sse4-2-and-avx-instructions

bazel build –config=opt –copt=-mavx –copt=-mavx2 –copt=-mfma –copt=-mfpmath=both –copt=-msse4.2 //tensorflow/tools/pip_package:build_pip_package

bazel build –config=opt –copt=-mavx –copt=-mavx2 –copt=-mfma //tensorflow/tools/pip_package:build_pip_package

Step 3

bazel-bin/tensorflow/tools/pip_package/build_pip_package /tmp/tensorflow_pkg
pip install /tmp/tensorflow_pkg/tensorflow-1.0.1-cp36-cp36m-macosx_10_7_x86_64.whl

Step 4

cd ~
ipython

Step 5

import tensorflow as tf
hello = tf.constant(‘Hello, TensorFlow!’)
sess = tf.Session()
print(sess.run(hello))

Step 6

pip uninstall /tmp/tensorflow_pkg/tensorflow-1.0.1-cp36-cp36m-macosx_10_7_x86_64.whl

Standard
Uncategorized

Building TensorFlow 1.1 from Source on mac osx. Sierra. macbook pro. i7. with SSE and AVX

/*

do each of these steps independently

will take around 1hr to complete all the steps regardless of machine type

*/

Step 0

sudo -i

Step 1

cd /
brew install bazel
pip install six numpy wheel

sudo apt-get install python3-numpy python3-dev python3-pip python3-wheel
rm -rf tensorflow

pip install –upgrade https://storage.googleapis.com/tensorflow/mac/cpu/protobuf-3.1.0-cp35-none-macosx_10_11_x86_64.whl

cd /
git clone https://github.com/tensorflow/tensorflow
cd /tensorflow
git checkout r1.0
cd /
chmod -R 777 tensorflow
cd /tensorflow
./configure # accept all default settings

Step 2

https://stackoverflow.com/questions/41293077/how-to-compile-tensorflow-with-sse4-2-and-avx-instructions

bazel build –config=opt –copt=-mavx –copt=-mavx2 –copt=-mfma –copt=-mfpmath=both –copt=-msse4.2 //tensorflow/tools/pip_package:build_pip_package

bazel build –config=opt –copt=-mavx –copt=-mavx2 –copt=-mfma //tensorflow/tools/pip_package:build_pip_package

Step 3

bazel-bin/tensorflow/tools/pip_package/build_pip_package /tmp/tensorflow_pkg
pip install /tmp/tensorflow_pkg/tensorflow-1.0.1-cp36-cp36m-macosx_10_7_x86_64.whl

Step 4

exit
cd ~
ipython

Step 5

import tensorflow as tf
hello = tf.constant(‘Hello, TensorFlow!’)
sess = tf.Session()
print(sess.run(hello))

Step 6

pip uninstall /tmp/tensorflow_pkg/tensorflow-1.3.0rc1-cp36-cp36m-macosx_10_7_x86_64.whl

Standard
Uncategorized

Building TensorFlow 1.1 from Source on mac osx. Sierra. macbook pro. i7.

Assumes only python3 is installed.

 

do each of these steps independently

Step 1

brew install bazel
sudo pip install six numpy wheel

sudo apt-get install python3-numpy python3-dev python3-pip python3-wheel
sudo rm -rf tensorflow

pip install –upgrade https://storage.googleapis.com/tensorflow/mac/cpu/protobuf-3.1.0-cp35-none-macosx_10_11_x86_64.whl

cd /
sudo git clone https://github.com/tensorflow/tensorflow
cd /tensorflow
sudo git checkout r1.0
cd /
sudo chmod -R 777 tensorflow
cd /tensorflow
./configure

Step 2

bazel build –config=opt //tensorflow/tools/pip_package:build_pip_package

Step 3

bazel-bin/tensorflow/tools/pip_package/build_pip_package /tmp/tensorflow_pkg
pip install /tmp/tensorflow_pkg/tensorflow-1.0.1-cp36-cp36m-macosx_10_7_x86_64.whl

Step 4

cd ~
ipython

Step 5

import tensorflow as tf
hello = tf.constant(‘Hello, TensorFlow!’)
sess = tf.Session()
print(sess.run(hello))

Step 6

pip uninstall /tmp/tensorflow_pkg/tensorflow-1.3.0rc1-cp36-cp36m-macosx_10_7_x86_64.whl

 

Standard