udacity

Machine Learning & Data Science

Linear Digressions - Old to New

Linear Digressions

Machine learning and particle physics go together like peanut butter and jelly--but this is a relatively new development.  

For many decades, physicists looked through their fairly large datasets using the laws of physics to guide their exploration; that tradition continues today, but as ever-larger datasets get made, machine learning becomes a more tractable way to deal with the deluge.  

With this in mind, ATLAS (one of the major experiments at CERN, the European Center for Nuclear Research and home laboratory of the recently discovered Higgs boson) ran a machine learning contest over the summer, to see what advances could be found from opening up the dataset to non-physicists.  

The results were impressive--physicists are smart folks, but there’s clearly lots of advances yet to make as machine learning and physics learn from one another.  And who knows--maybe more Nobel prizes to win as well!

https://www.kaggle.com/c/higgs-boson

Hunting for the Higgs

In the 1850s, there were a lot of things we didn’t know yet: how to create an airplane, how to split an atom, or how to control the spread of a common but deadly disease: cholera.  

When a cholera outbreak in London killed scores of people, a doctor named John Snow used it as a chance to study whether the cause might be very small organisms that were spreading through the water supply (the prevailing theory at the time was miasma, or “bad air”).  By tracing the geography of all the deaths from the outbreak, Snow was practicing elementary data science--and stumbled upon one of history’s most famous outliers.  

In this episode, we’ll tell you more about this single data point, a case of cholera that cracked the case wide open for Snow and provided critical validation for the germ theory of disease.

http://en.wikipedia.org/wiki/1854_Broad_Street_cholera_outbreak

How Outliers Helped Defeat Cholera

Computers are capable of many impressive feats, but making you laugh is usually not one of them. Or could it be? This episode will talk about a custom-built machine learning algorithm that searches through text and writes jokes based on what it finds. 

The jokes are formulaic: they're all of the form "I like my X like I like my Y: Z" where X and Y are nouns, and Z is an adjective that can describe both X and Y. For (dumb) example, "I like my men like I like my coffee: steaming hot." The joke is funny when ZX and ZY are both very common phrases, but X and Y are rarely seen together. 

So, given a large enough corpus of text, the algorithm looks for triplets of words that fit this description and writes jokes based on them. Are the jokes funny? You be the judge...

http://homepages.inf.ed.ac.uk/s0894589/petrovic13unsupervised.pdf

Computers Try to Tell Jokes

Baseball is characterized by a high level of equality between teams; even the best teams might only have 55% win percentages (contrast this with college football, where teams go undefeated pretty regularly). In this regime, where 2 outcomes (Giants win/Giants lose) are approximately equally likely, we can model the win/loss chances with a binomial distribution. 

Using the binomial distribution, we can calculate an interesting little result: what's the chance of the world series going to only 4 games? 5? 6? All the way to 7? Then we can compare to decades' worth of world series data, to see how well the data follows the binomial assumption. 

The result tells us a lot about sports psychology--if each game is independent of the others, 4/5/6/7 game series are equally likely. The data shows a different trend: 4 and 7 game series are significantly more likely than 5 or 6. There's a powerful psychological effect at play--everybody loves the 7th game of the world series, or a good sweep. And it turns out that the baseball teams, whether they intend it or not, oblige our love of short (4) and long (7) world series!

http://blog.philbirnbaum.com/2007/06/winning-world-series-in-x-games.html

Stats of World Series Streaks

A true classic topic in ML: Facial recognition is very high-dimensional, meaning that each picture can have millions of pixels, each of which can be a single feature. It's computationally expensive to deal with all these features, and invites overfitting problems. PCA (principal components analysis) is a classic dimensionality reduction tool that compresses these many dimensions into the few that contain the most variation in the data, and those principal components are often then fed into a classic ML algorithm like and SVM. 

One of the best thing about eigenfaces is the great example code that you can find in sklearn--you can distinguish pictures of world leaders yourself in just a few minutes!

http://scikit-learn.org/stable/auto_examples/applications/face_recognition.html

Facial Recognition with Eigenfaces

Now that we know about eigenfaces (if you don't, listen to the previous episode), let's talk about how it breaks down. 

Variations that are trivial to humans when identifying faces can really mess up computer-driven facial ID--expressions, lighting, and angle are a few. Something that can easily happen is an algorithm can optimize to identify one of those traits, rather than the underlying trait of whether the person is the same (for example, if the training image is me smiling, you may reject an image of me frowning but accidentally approve an image of another woman smiling). 

Fisherfaces uses a fisher linear discriminant to distinguish based on the dimension in the data that shows the smallest inter-class distance, rather than maximizing the variation overall (we'll unpack this statement), and it is much more robust than our pal eigenfaces when there's shadows, cut-off images, expressions, etc.

http://www.cs.columbia.edu/~belhumeur/journal/fisherface-pami97.pdf

Better Facial Recognition with Fisherfaces

So, um... what about machine learning for audio applications?  In the course of starting this podcast, we've edited out a lot of "um"'s from our raw audio files.  It's gotten now to the point that, when we see the waveform in soundstudio, we can almost identify an "um" by eye.  Which makes it an interesting problem for machine learning--is there a way we can train an algorithm to recognize the "um" pattern, too?  This has become a little side project for Katie, which is very much still a work in progress.  We'll talk about what's been accomplished so far, some design choices Katie made in getting the project off the ground, and (of course) mistakes made and hopefully corrected.  We always say that the best way to learn something is by doing it, and this is our chance to try our own machine learning project instead of just telling you about what someone else did! 

Um Detector 1

Supervised classification is built on the backs of labeled datasets, but a good set of labels can be hard to find.  Great data is everywhere, but the corresponding labels can sometimes be really tricky.  Take a few examples we've already covered, like lie detection with an MRI machine (have to take pictures of someone's brain while they try to lie, not a trivial task) or automated image captioning (so many images!  so many valid labels!)  

In this epsiode, we'll dig into this topic in depth, talking about some of the standard ways to get a labeled dataset if your project requires labels and you don't already have them.

www.higgshunters.org

Labels and Where To Find Them

Often machine learning discussions center around algorithms, or features, or datasets--this one centers around interpretation, and ethics. 

Suppose you could use a technology like fMRI to see what regions of a person's brain are active when they ask questions. And also suppose that you could run trials where you watch their brain activity while they lie about some minor issue (say, whether the card in their hand is a spade or a club)--could you use machine learning to analyze those images, and use the patterns in them for lie detection? Well you certainly can try, and indeed researchers have done just that. 

There are important problems though--the images of brains can be high variance, meaning that for any given person, there might not be a lot of certainty about whether they're lying or not. It's also open to debate whether the training set (in this case, test subjects with playing cards in their hands) really generalize well to the more important cases, like a person accused of a crime. 

So while machine learning has yielded some impressive gains in lie detection, it is not a solution to these thornier scientific issues.

http://www.amacad.org/pdfs/deceit.pdf

Lie Detectors

Ray Kurzweil‘s impact on my life in general but especially on what I have been doing for the past 3 or 4 years is hard to exaggerate. It is a simple fact that, if I haven’t read his seminal book The Singularity is Near, I would be neither blogging nor podcasting about exponential technologies, not to mention going to Singularity University. And so it was with great excitement and some trepidation that I went to interview Dr. Kurzweil in his office in Boston.

Part of my trepidation came from some technical concerns: I wish I could buy a better camera. I wish I could hire a team of audio and video professionals so that I can focus on the interview itself. I wish I did a better job with the set up. I wish I had noticed that Ray’s lavalier mic has slipped out of its holder… The list is exponential. Still, if there is one thing that I’ve learned since I started podcasting is that we do get better. But it takes time. Meanwhile, my consolation is that eventually I will have to do another interview with Ray Kurzweil just so I get it right that time. For now, however, I hope that the content of this one will make up for its technical deficiencies.

During our conversation with Dr. Kurzweil we cover a wide variety of topics such as: how and why at age 5 Ray decided to become an inventor; his unique background of being born to Jewish parents but brought up in a Unitarian Church; his early interest in issues such as religious tolerance, poverty, social inequality and justice; 3D printing, open source, patents, progress and intellectual property rights; Watson, artificial intelligence, the Turing Test and human rights for AI, the technological singularity and some criticism thereof; his upcoming book How To Create A Mind and his Pattern Recognition Theory of Mind; the evolutionary advantages of intelligence; the benefits of reverse-engineering the human brain for the creation of AI and whether the latter would be interested in pondering and solving humanity’s greatest problems.

My favorite quote that I will take away from this interview with Ray Kurzweil is:

“Don’t be too concerned about what’s practical. Follow your passion and be who you would like to be.”

Ray Kurzweil on the Singularity: Be Who You Would Like To Be

Gary Marcus is not only a professor in psychology but also a computer scientist, programmer, AI researcher and best selling author. He recently wrote a critical article titled Ray Kurzweil’s Dubious New Theory of Mind that was publsihed by The New Yorker. After reading his blog post I thought that it will be interesting to invite Gary on Singularity 1 on 1 so that he can talk more about his argument.   

During our conversation with Gary we cover a wide variety of topics such as: what is psychology and how he got interested in it; his theory of mind in general and the idea that the mind is a kluge in particular; why the best approach to creating AI is at the bridge between neuroscience and psychology; other projects such as Henry Markram‘s Blue Brain, Randal Koene‘s Whole Brain Emulation, and Dharmendra Modha‘s SyNAPSE; Ray Kurzweil’s Patern Recognition Theory of Mind; Deep Blue, Watson and the lessons thereof; his take on the technological singularity and the ethics surrounding the creation of friendly AI…

Gary Marcus: How do we bridge the mind with the brain?!…

O'Reilly's Ben Lorica chats with Mike Cafarella, assistant professor of computer science at the University of Michigan and co-founder of both Hadoop and Nutch.

From Search to Distributed Computing to Large-scale Information Extraction

In this special holiday episode of the O’Reilly Data Show, I look back at two conversations I had earlier this year at the Spark Summit in San Francisco. The first segment is an on-stage fireside chat with Ben Horowitz, co-founder of Andreessen Horowitz and author of The Hard Thing About Hard Things.

In the second segment, Reynold Xin, one of the architects of Apache Spark, explains the rise of Apache Spark in China.

Investing in Big Data Technologies

The reason that neural nets are taking over the world right now is because they can be efficiently trained with the backpropagation algorithm.  In short, backprop allows you to adjust the weights of the neural net based on how good of a job the neural net is doing at classifying training examples, thereby getting better and better at making predictions.  In this episode: we talk backpropagation, and how it makes it possible to train the neural nets we know and love.

Backpropagation

An interview with Chris Dyer.

https://www.semanticscholar.org/paper/Recurrent-Neural-Network-Grammars-Dyer-Kuncoro/1594d954abc650bce2db445c52a76e49655efb0c

04 - Recurrent Neural Network Grammars, with Chris Dyer

Open source software is ubiquitous throughout data science, and enables the work of nearly every data scientist in some way or another. Open source projects, however, are disproportionately maintained by a small number of individuals, some of whom are institutionally supported, but many of whom do this maintenance on a purely volunteer basis. The health of the data science ecosystem depends on the support of open source projects, on an individual and institutional level.

https://hdsr.mitpress.mit.edu/pub/xsrt4zs2/release/2

The Data Science Open Source Ecosystem

This is a re-release of an episode that originally ran in October 2019.

If you’re trying to manage a project that serves up analytics data for a few very distinct uses, you’d be wise to consider having custom solutions for each use case that are optimized for the needs and constraints of that use cases. You also wouldn’t be YouTube, which found themselves with this problem (gigantic data needs and several very different use cases of what they needed to do with that data) and went a different way: they built one analytics data system to serve them all. Procella, the system they built, is the topic of our episode today: by deconstructing the system, we dig into the four motivating uses of this system, the complexity they had to introduce to service all four uses simultaneously, and the impressive engineering that has to go into building something that “just works.”

Procella: YouTube's super-system for analytics data storage

A few weeks ago, we put out a call for data scientists interested in issues of race and racism, or people studying how those topics can be studied with data science methods, should get in touch to come talk to our audience about their work. This week we’re excited to bring on Todd Hendricks, Bay Area data scientist and a volunteer who reached out to tell us about his studies with the Stanford Open Policing dataset.

A Data Science Take on Open Policing Data

The data science and artificial intelligence community has made amazing strides in the past few years to algorithmically automate portions of the healthcare process. This episode looks at two computer vision algorithms, one that diagnoses diabetic retinopathy and another that classifies liver cancer, and asks the question—are patients now getting better care, and achieving better outcomes, with these algorithms in the mix? The answer isn’t no, exactly, but it’s not a resounding yes, because these algorithms interact with a very complex system (the healthcare system) and other shortcomings of that system are proving hard to automate away. Getting a faster diagnosis from an image might not be an improvement if the image is now harder to capture (because of strict data quality requirements associated with the algorithm that wouldn’t stop a human doing the same job). Likewise, an algorithm getting a prediction mostly correct might not be an overall benefit if it introduces more dramatic failures when the prediction happens to be wrong. For every data scientist whose work is deployed into some kind of product, and is being used to solve real-world problems, these papers underscore how important and difficult it is to consider all the context around those problems.

A Reality Check on AI-Driven Medical Assistants

All good things must come to an end, including this podcast. This is the last episode we plan to release, and it doesn’t cover data science—it’s mostly reminiscing, thanking our wonderful audience (that’s you!), and marveling at how this thing that started out as a side project grew into a huge part of our lives for over 5 years.

It’s been a ride, and a real pleasure and privilege to talk to you each week. Thanks, best wishes, and good night!

—Katie and Ben