Ai & ml

Linear Digressions

Data science

One tricky thing about working with time series data, like the audio data in our "um" detector (remember that?  because we barely do...), is that sometimes events look really similar but one is a little bit stretched and squeezed relative to the other.  Besides having an amazing name, the dynamic time warp is a handy algorithm for aligning two time series sequences that are close in shape, but don't quite line up out of the box.

Relevant link:
http://www.aaai.org/Papers/Workshops/1994/WS-94-03/WS94-03-031.pdf

Um Detector 2: The Dynamic Time Warp

In episode sixteen of season two, we get an introduction to Restricted Boltzmann Machines, we take a listener question about tuning hyperparameters,  plus we talk with Eric Lander of the Broad Institute. 

Eric Lander and Restricted Boltzmann Machines

In this session of the Super Data Science Podcast, I chat with machine learning expert Hadelin de Ponteves about his work at Google, Canal+ and what career implications may come from the rapid growth of the Data Science field.

If you enjoyed this episode, check out show notes, resources, and more at https://www.superdatascience.com/2

The Super Data Science Podcast: Big Data | Analytics Careers | Mentors | Success

SDS 002 : Machine Learning, Recommender Systems and The Future of Data with Hadelin de Ponteves

Tobias - Ternström

Picking a metric for a problem means defining how you’ll measure success in solving that problem. Which sounds important, because it is, but oftentimes new data scientists only get experience with a few kinds of metrics when they’re learning and those metrics have real shortcomings when you think about what they tell you, or don’t, about how well you’re really solving the underlying problem. This episode takes a step back and says, what are some metrics that are popular with data scientists, why are they popular, and what are their shortcomings when it comes to the real world? There’s been a lot of great thinking and writing recently on this topic, and we cover a lot of that discussion along with some perspective of our own.

Relevant links:
https://www.fast.ai/2019/09/24/metrics/
https://arxiv.org/abs/1909.12475
https://medium.com/shoprunner/evaluating-classification-models-1-ff0730801f17
https://hbr.org/2019/09/dont-let-metrics-undermine-your-business

Data scientists: beware of simple metrics

AI is evolving incredibly quickly, and thinking now about where it might go next (and how we as a species and a society should be prepared) is critical. Professor Stuart Russell, an AI expert at UC Berkeley, has a formulation for modifications to AI that we should study and try implementing now to keep it much safer in the long run. Prof. Russell’s new book, “Human Compatible: Artificial Intelligence and the Problem of Control” gives an accessible but deeply thoughtful exploration of why he thinks runaway AI is something we need to be considering seriously now, and what changes in formulation might be a solution. This episodes features Prof. Russell as a special guest, exploring the topics in his book and giving more perspective on the long-term possible futures of AI: both good and bad.

Relevant links:
https://www.penguinrandomhouse.com/books/566677/human-compatible-by-stuart-russell/

Changing our formulation of AI to avoid runaway risks: Interview with Prof. Stuart Russell

The abundance of data in healthcare, and the value we could capture from structuring and analyzing that data, is a huge opportunity. It also presents huge challenges. One of the biggest challenges is how, exactly, to do that structuring and analysis—data scientists working with this data have hundreds or thousands of small, and sometimes large, decisions to make in their day-to-day analysis work. What data should they include in their studies? What method should they use to analyze it? What hyperparameter settings should they explore, and how should they pick a value for their hyperparameters? The thing that’s really difficult here is that, depending on which path they choose among many reasonable options, a data scientist can get really different answers to the underlying question, which makes you wonder how to conclude anything with certainty at all.

The paper for this week’s episode performs a systematic study of many, many different permutations of the questions above on a set of benchmark datasets where the “right” answers are known. Which strategies are most likely to yield the “right” answers? That’s the whole topic of discussion.

Relevant links:
https://hdsr.mitpress.mit.edu/pub/fxz7kr65

Keeping ourselves honest when we work with observational healthcare data

A message from Ben around algorithmic bias, and how our models are sometimes reflections of ourselves.

An interstitial word from Ben

Open source software is ubiquitous throughout data science, and enables the work of nearly every data scientist in some way or another. Open source projects, however, are disproportionately maintained by a small number of individuals, some of whom are institutionally supported, but many of whom do this maintenance on a purely volunteer basis. The health of the data science ecosystem depends on the support of open source projects, on an individual and institutional level.

https://hdsr.mitpress.mit.edu/pub/xsrt4zs2/release/2

The Data Science Open Source Ecosystem

This is a re-release of an episode that originally ran in October 2019.

If you’re trying to manage a project that serves up analytics data for a few very distinct uses, you’d be wise to consider having custom solutions for each use case that are optimized for the needs and constraints of that use cases. You also wouldn’t be YouTube, which found themselves with this problem (gigantic data needs and several very different use cases of what they needed to do with that data) and went a different way: they built one analytics data system to serve them all. Procella, the system they built, is the topic of our episode today: by deconstructing the system, we dig into the four motivating uses of this system, the complexity they had to introduce to service all four uses simultaneously, and the impressive engineering that has to go into building something that “just works.”

Procella: YouTube's super-system for analytics data storage

A few weeks ago, we put out a call for data scientists interested in issues of race and racism, or people studying how those topics can be studied with data science methods, should get in touch to come talk to our audience about their work. This week we’re excited to bring on Todd Hendricks, Bay Area data scientist and a volunteer who reached out to tell us about his studies with the Stanford Open Policing dataset.

A Data Science Take on Open Policing Data

The data science and artificial intelligence community has made amazing strides in the past few years to algorithmically automate portions of the healthcare process. This episode looks at two computer vision algorithms, one that diagnoses diabetic retinopathy and another that classifies liver cancer, and asks the question—are patients now getting better care, and achieving better outcomes, with these algorithms in the mix? The answer isn’t no, exactly, but it’s not a resounding yes, because these algorithms interact with a very complex system (the healthcare system) and other shortcomings of that system are proving hard to automate away. Getting a faster diagnosis from an image might not be an improvement if the image is now harder to capture (because of strict data quality requirements associated with the algorithm that wouldn’t stop a human doing the same job). Likewise, an algorithm getting a prediction mostly correct might not be an overall benefit if it introduces more dramatic failures when the prediction happens to be wrong. For every data scientist whose work is deployed into some kind of product, and is being used to solve real-world problems, these papers underscore how important and difficult it is to consider all the context around those problems.

A Reality Check on AI-Driven Medical Assistants

All good things must come to an end, including this podcast. This is the last episode we plan to release, and it doesn’t cover data science—it’s mostly reminiscing, thanking our wonderful audience (that’s you!), and marveling at how this thing that started out as a side project grew into a huge part of our lives for over 5 years.

It’s been a ride, and a real pleasure and privilege to talk to you each week. Thanks, best wishes, and good night!

—Katie and Ben