nextup

linear digression

Linear Digressions

misc

The Medici were one of the great ruling families of Europe during the Renaissance.  How did they come to rule?  Not power, or money, or armies, but through the strength of their social network.  And speaking of great historical social networks, analysis of the network of letter-writing during the Enlightenment is helping humanities scholars track the dispersion of great ideas across the world during that time, from Voltaire to Benjamin Franklin and everyone in between.

Relevant links:
https://www2.bc.edu/~jonescq/mb851/Mar12/PadgettAnsell_AJS_1993.pdf
http://republicofletters.stanford.edu/index.html

Great Social Networks in History

Survival analysis is all about studying how long until an event occurs--it's used in marketing to study how long a customer stays with a service, in epidemiology to estimate the duration of survival of a patient with some illness, and in social science to understand how the characteristics of a war inform how long the war goes on.  This episode talks about the special challenges associated with survival analysis, and the tools that (data) scientists use to answer all kinds of duration-related questions.

Survival Analysis

First up in this episode: a crash course in natural language processing, and important steps if you want to use machine learning techniques on text data.  Then we'll take that NLP know-how and talk about a really cool analysis of State of the Union text, which analyzes the topics and word choices of every President from Washington to Obama.

Relevant link:
https://civisanalytics.com/blog/data-science/2016/01/15/data-science-on-state-of-the-union-addresses/

Text Analysis on the State Of The Union

"People with a family history of heart disease are more likely to eat healthy foods, and have a high incidence of heart attacks."  Did the healthy food cause the heart attacks?  Probably not.  But establishing causal links is extremely tricky, and extremely important to get right if you're trying to help students, test new medicines, or just optimize a website.  In this episode, we'll unpack randomized experiments, like AB tests, and maybe you'll be smarter as a result.  Will you be smarter BECAUSE of this episode?  Well, tough to say for sure...

Relevant link:
http://tylervigen.com/spurious-correlations

Experiments and Messy, Tricky Causality

This episode started out as a discussion of recommendation engines, like Netflix uses to suggest movies.  There's still a lot of that in here.  But a related topic, which is both interesting and important, is how to keep data private in the era of large-scale recommendation engines--what mistakes have been made surrounding supposedly anonymized data, how data ends up de-anonymized, and why it matters for you.

Relevant links:
http://www.netflixprize.com/
http://bits.blogs.nytimes.com/2010/03/12/netflix-cancels-contest-plans-and-settles-suit/?_r=0
http://arxiv.org/PS_cache/cs/pdf/0610/0610105v2.pdf

Recommendation Engines and Privacy

Competing in a machine learning competition on Kaggle is a kind of rite of passage for data scientists.  Losing unexpectedly at the very end of the contest is also something that a lot of us have experienced.  It's not just bad luck: a very specific combination of overfitting on popular competitions can take someone who is in the top few spots in the final days of a contest and bump them down hundreds of slots in the final tally.

How to Lose at Kaggle

This week: everybody's favorite WWII-era classifier metric!  But it's not just for winning wars, it's a fantastic go-to metric for all your classifier quality needs.

Rock the ROC Curve

Melvyn Bragg and guests discuss Seneca the Younger, who was one of the first great writers to live his entire life in the world of the new Roman empire, after the fall of the Republic. He was a Stoic philosopher, he wrote blood-soaked tragedies, he was an orator, and he navigated his way through the reigns of Caligula, Claudius and Nero, sometimes exercising power at the highest level and at others spending years in exile. Agrippina the Younger was the one who called for him to tutor Nero, and it is thought Seneca helped curb some of Nero's excesses. He was later revered within the Christian church, partly for what he did and partly for what he was said to have done in forged letters to St Paul. His tragedies, with their ghosts and high body count, influenced Shakespeare's Titus Andronicus and Hamlet, and Kyd's Spanish Tragedy. The image above is the so-called bust of Seneca, a detail from Four Philosophers by Peter Paul Rubens.

With

Mary Beard
Professor of Classics at the University of Cambridge

Catharine Edwards
Professor of Classics and Ancient History at Birkbeck, University of London

and

Alessandro Schiesaro
Professor of Classics at the University of Manchester

Producer: Simon Tillotson. Feb 23, 2017

Seneca the Younger

This week's episode is just in time for JupyterCon in NYC, August 22-25...

Jupyter notebooks are probably familiar to a lot of data nerds out there as a great open-source tool for exploring data, doing quick visualizations, and packaging code snippets with explanations for sharing your work with others.  If you're not a data person, or you are but you haven't tried out Jupyter notebooks yet, here's your nudge to go give them a try.  In this episode we'll go back to the old days, before notebooks, and talk about all the ways that data scientists like to work that wasn't particularly well-suited to the command line + text editor setup, and talk about how notebooks have evolved over their lifetime to become even more powerful and well-suited to the data scientist's workflow.

Jupyter Notebooks

As data science matures as a field, it's becoming clearer what attributes a data science team needs to have to elevate their work to the next level.  Most of our episodes are about the cool work being done by other people, but this one summarizes some thinking Katie's been doing herself around how to guide data science teams toward more mature, effective practices.  We'll go through five key characteristics of great data science teams, which we collectively refer to as "disciplined data science," and why they matter.

Disciplined Data Science

The Fourier transform is one of the handiest tools in signal processing for dealing with periodic time series data. Using a Fourier transform, you can break apart a complex periodic function into a bunch of sine and cosine waves, and figure out what the amplitude, frequency and offset of those component waves are. It's a really handy way of re-expressing periodic data--you'll never look at a time series graph the same way again.

The Fourier Transform

If you've been in data science for more than a year or two, chances are you've noticed changes in the field as it's grown and matured. And if you're newer to the field, you may feel like there's a disconnect between lots of different stories about what data scientists should know, or do, or expect from their job. This week, we cover two thought pieces, one that arose from interviews with 35(!) data scientists speaking about what their jobs actually are (and aren't), and one from the head of data science at AirBnb organizing core data science work into three main specialties.

Relevant links:
https://hbr.org/2018/08/what-data-scientists-really-do-according-to-35-data-scientists
https://www.linkedin.com/pulse/one-data-science-job-doesnt-fit-all-elena-grewal

The Three Types of Data Scientists, and What They Actually Do

Speaker(s): Professor David Spiegelhalter | In his new book, The Art of Statistics, David Spiegelhalter guides us through the essential principles we need in order to derive knowledge from data, showing us why data can never speak for itself. He explains the basic concepts, from regression to P-values (without using mathematics), and introduces the intellectual ideas that underpin statistics. Drawing on numerous real world examples, he shows us how statistics can help us determine the luckiest passenger on the Titanic, whether serial killer Harold Shipman could have been caught earlier, and if the skeleton in the Leicester car park really was Richard III. Sir David Spiegelhalter is a British statistician and Chair of the Winton Centre for Risk and Evidence Communication in the Statistical Laboratory at the University of Cambridge. Spiegelhalter is one of the most cited and influential researchers in his field, and was elected as President of the Royal Statistical Society for 2017-18. Fiona Steele is a Professor of Statistics and Deputy Head of the Department of Statistics at LSE. Fiona first joined in LSE in 1996 as Lecturer in Statistics and Research Methodology.  She then worked at the Institute of Education, University of London 2001-2005, followed by the University of Bristol 2005-2013 where she was Professor of Social Statistics and Director of the Centre for Multilevel Modelling. She returned to LSE in 2013. The Department of Statistics (@StatsDeptLSE) offers a vibrant research environment and a comprehensive programme of undergraduate and postgraduate degrees.

Learning from Data: the art of statistics [Audio]

Open source software is ubiquitous throughout data science, and enables the work of nearly every data scientist in some way or another. Open source projects, however, are disproportionately maintained by a small number of individuals, some of whom are institutionally supported, but many of whom do this maintenance on a purely volunteer basis. The health of the data science ecosystem depends on the support of open source projects, on an individual and institutional level.

https://hdsr.mitpress.mit.edu/pub/xsrt4zs2/release/2

The Data Science Open Source Ecosystem

This is a re-release of an episode that originally ran in October 2019.

If you’re trying to manage a project that serves up analytics data for a few very distinct uses, you’d be wise to consider having custom solutions for each use case that are optimized for the needs and constraints of that use cases. You also wouldn’t be YouTube, which found themselves with this problem (gigantic data needs and several very different use cases of what they needed to do with that data) and went a different way: they built one analytics data system to serve them all. Procella, the system they built, is the topic of our episode today: by deconstructing the system, we dig into the four motivating uses of this system, the complexity they had to introduce to service all four uses simultaneously, and the impressive engineering that has to go into building something that “just works.”

Procella: YouTube's super-system for analytics data storage

A few weeks ago, we put out a call for data scientists interested in issues of race and racism, or people studying how those topics can be studied with data science methods, should get in touch to come talk to our audience about their work. This week we’re excited to bring on Todd Hendricks, Bay Area data scientist and a volunteer who reached out to tell us about his studies with the Stanford Open Policing dataset.

A Data Science Take on Open Policing Data

The data science and artificial intelligence community has made amazing strides in the past few years to algorithmically automate portions of the healthcare process. This episode looks at two computer vision algorithms, one that diagnoses diabetic retinopathy and another that classifies liver cancer, and asks the question—are patients now getting better care, and achieving better outcomes, with these algorithms in the mix? The answer isn’t no, exactly, but it’s not a resounding yes, because these algorithms interact with a very complex system (the healthcare system) and other shortcomings of that system are proving hard to automate away. Getting a faster diagnosis from an image might not be an improvement if the image is now harder to capture (because of strict data quality requirements associated with the algorithm that wouldn’t stop a human doing the same job). Likewise, an algorithm getting a prediction mostly correct might not be an overall benefit if it introduces more dramatic failures when the prediction happens to be wrong. For every data scientist whose work is deployed into some kind of product, and is being used to solve real-world problems, these papers underscore how important and difficult it is to consider all the context around those problems.

A Reality Check on AI-Driven Medical Assistants

All good things must come to an end, including this podcast. This is the last episode we plan to release, and it doesn’t cover data science—it’s mostly reminiscing, thanking our wonderful audience (that’s you!), and marveling at how this thing that started out as a side project grew into a huge part of our lives for over 5 years.

It’s been a ride, and a real pleasure and privilege to talk to you each week. Thanks, best wishes, and good night!

—Katie and Ben