DataSciPodcast

After the Challenger exploded in 1986, killing all 7 astronauts aboard, an investigation into the cause was immediately launched. 

In the cold temperatures the night before the launch, the o-rings that seal off the fuel tanks from the rocket boosters became inflexible, so they did not seal properly, which led to the fuel tank explosion. NASA knew that there could be o-ring problems, but performed the analysis of their data incorrectly and ended up massively underestimating the risk associated with the cold temperatures. 

In this episode, we'll unpack the mistakes they made. We'll talk about how they excluded data points that they thought were irrelevant but which actually were critical to recognizing a fatal pattern.

Statistical Mistakes and the Challenger Disaster

Let’s talk money. As a “hot” career right now, data science can pay pretty well. But for an individual person matched with a specific job or industry, how much should someone expect to make? 

Since Katie was on the job market lately, this was something she’s been researching, and it turns out that data science itself (in particular linear regressions) has some answers. 

In this episode, we go through a survey of hundreds of data scientists, who report on their job duties, industry, skills, education, location, etc. along with their salaries, and then talk about how this data was fed into a linear regression so that you (yes, you!) can use the patterns in the data to know what kind of salary any particular kind of data scientist might expect.

Careers in Data Science

Let's talk about randomness! Although randomness is pervasive throughout the natural world, it's surprisingly difficult to generate random numbers. And even if your numbers look random (but actually aren't), it can have interesting consequences on the security of systems, and the accuracy of models and research. 

In this episode, Katie and Ben talk about randomness, its place in machine learning and computation in general, along with some random digressions of their own.

Random Number Generation

Come get a little "out there" with us this week, as we use a meta-study of extrasensory perception (or ESP, often used in the same sentence as "psychics") to chat about Bayesian vs. frequentist statistics.

Bayesian Psychics

When you sleep, the neural pathways in your brain take the "white noise" of your resting brain, mix in your experiences and imagination, and the result is dreams (that is a highly unscientific explanation, but you get the idea).  What happens when neural nets are put through the same process?  Train a neural net to recognize pictures, and then send through an image of white noise, and it will start to see some weird (but cool!) stuff.

Neural Net Inception

Overfitting to your training data can be avoided by evaluating your machine learning algorithm on a holdout test dataset, but what about overfitting to the test data?  Turns out it can be done, easily, and you have to be very careful to avoid it.  But an algorithm from the field of privacy research shows promise for keeping your test data safe from accidental overfitting

Thresholdout: Down with Overfitting

Because there are more interesting problems than there are labeled datasets, semi-supervised learning provides a framework for getting feedback from the environment as a proxy for labels of what's "correct."  Of all the machine learning methodologies, it might also be the closest to how humans usually learn--we go through the world, getting (noisy) feedback on the choices we make and learn from the outcomes of our actions.  

A Criminally Short Introduction to Semi Supervised Learning

Grab a cocktail, put on your favorite karaoke track, and let’s talk some more about disentangling audio data!

The Cocktail Party Problem

Machine learning: it can be fooled, just like you or me.  Here's one of our favorite examples, a study into hacking neural networks.

Relevant links:
http://arxiv.org/pdf/1412.1897v4.pdf

Hacking Neural Nets

In order to do supervised learning, you need a labeled training dataset.  Or do you...?

Relevant links:
http://www.cs.columbia.edu/~dplewis/candidacy/goldman00enhancing.pdf

Unlabeled Supervised Learning--whaaa?

Psychometrics is all about measuring the psychological characteristics of people; for example, scholastic aptitude.  How is this done?  Tests, of course!  But there's a chicken-and-egg problem here: you need to know both how hard a test is, and how smart the test-taker is, in order to get the results you want.  How to solve this problem, one equation with two unknowns?  Item response theory--the data science behind such tests and the GRE.

Relevant links: 
https://en.wikipedia.org/wiki/Item_response_theory

Item Response Theory: how smart ARE you?

Let's imagine a future in which a truly intelligent computer program exists.  How would it convince us (humanity) that it was intelligent?  Alan Turing's answer to this question, proposed over 60 years ago, is that the program could convince a human conversational partner that it, the computer, was in fact a human.  60 years later, the Turing Test endures as a gold standard of artificial intelligence.  It hasn't been beaten, either--yet.

Relevant links:
https://en.wikipedia.org/wiki/Turing_test
http://commonsensereasoning.org/winograd.html
http://consumerist.com/2015/09/29/its-not-just-you-robots-are-also-bad-at-assembling-ikea-furniture/

The Turing Test

Survival analysis is all about studying how long until an event occurs--it's used in marketing to study how long a customer stays with a service, in epidemiology to estimate the duration of survival of a patient with some illness, and in social science to understand how the characteristics of a war inform how long the war goes on.  This episode talks about the special challenges associated with survival analysis, and the tools that (data) scientists use to answer all kinds of duration-related questions.

Survival Analysis

"People with a family history of heart disease are more likely to eat healthy foods, and have a high incidence of heart attacks."  Did the healthy food cause the heart attacks?  Probably not.  But establishing causal links is extremely tricky, and extremely important to get right if you're trying to help students, test new medicines, or just optimize a website.  In this episode, we'll unpack randomized experiments, like AB tests, and maybe you'll be smarter as a result.  Will you be smarter BECAUSE of this episode?  Well, tough to say for sure...

Relevant link:
http://tylervigen.com/spurious-correlations

Experiments and Messy, Tricky Causality

One neural net is creating counterfeit bills and passing them off to a second neural net, which is trying to distinguish the real money from the fakes.  Result: two neural nets that are better than either one would have been without the competition.

Relevant links:
http://arxiv.org/pdf/1406.2661v1.pdf
http://arxiv.org/pdf/1412.6572v3.pdf
http://soumith.ch/eyescream/

Neural nets play cops and robbers (AKA generative adverserial networks)

Sometimes when we say an algorithm discriminates, we mean it can tell the difference between two types of items.  But in this episode, we'll talk about another, more troublesome side to discrimination: algorithms can be... racist?  Sexist?  Ageist?  Yes to all of the above.  It's an important thing to be aware of, especially when doing people-centered data science.  We'll discuss how and why this happens, and what solutions are out there (or not).

Relevant Links:
http://www.nytimes.com/2015/07/10/upshot/when-algorithms-discriminate.html
http://techcrunch.com/2015/08/02/machine-learning-and-human-bias-an-uneasy-pair/
http://www.sciencefriday.com/segments/why-machines-discriminate-and-how-to-fix-them/
https://medium.com/@geomblog/when-an-algorithm-isn-t-2b9fe01b9bb5#.auxqi5srz




Discriminatory Algorithms

Machine learning algorithms can be black boxes--inputs go in, outputs come out, and what happens in the middle is anybody's guess.  But understanding how a model arrives at an answer is critical for interpreting the model, and for knowing if it's doing something reasonable (one could even say... trustworthy).  We'll talk about a new algorithm called LIME that seeks to make any model more understandable and interpretable.

Relevant Links:
http://arxiv.org/abs/1602.04938
https://github.com/marcotcr/lime/tree/master/lime

Model Interpretation (and Trust Issues)

Supervised machine learning assumes that the features and labels used for building a classifier are isolated from each other--basically, that you can't cheat by peeking.  Turns out this can be easier said than done.  In this episode, we'll talk about the many (and diverse!) cases where label information contaminates features, ruining data science competitions along the way.

Relevant links:
https://www.researchgate.net/profile/Claudia_Perlich/publication/221653692_Leakage_in_data_mining_Formulation_detection_and_avoidance/links/54418bb80cf2a6a049a5a0ca.pdf

Data Contamination

Conjoint analysis is like AB tester, but more bigger more better: instead of testing one or two things, you can test potentially dozens of options.  Where might you use something like this?  Well, if you wanted to design an entire hotel chain completely from scratch, and to do it in a data-driven way.  You'll never look at Courtyard by Marriott the same way again.

Relevant link: https://marketing.wharton.upenn.edu/files/?whdmsaction=public:main.file&fileID=466 

Conjoint Analysis: like AB testing, but on steroids

Apple wants to study iPhone users' activities and use it to improve performance.  Google collects data on what people are doing online to try to improve their Chrome browser.  Do you like the idea of this data being collected?  Maybe not, if it's being collected on you--but you probably also realize that there is some benefit to be had from the improved iPhones and web browsers.  Differential privacy is a set of policies that walks the line between individual privacy and better data, including even some old-school tricks that scientists use to get people to answer embarrassing questions honestly.  

Relevant links: http://static.googleusercontent.com/media/research.google.com/en//pubs/archive/42852.pdf

DataSciPodcast

Tyler Miguel 1