nextup

tech

linear digression

random

Linear Digressions

Data Science

Overfitting to your training data can be avoided by evaluating your machine learning algorithm on a holdout test dataset, but what about overfitting to the test data?  Turns out it can be done, easily, and you have to be very careful to avoid it.  But an algorithm from the field of privacy research shows promise for keeping your test data safe from accidental overfitting

Thresholdout: Down with Overfitting

The Medici were one of the great ruling families of Europe during the Renaissance.  How did they come to rule?  Not power, or money, or armies, but through the strength of their social network.  And speaking of great historical social networks, analysis of the network of letter-writing during the Enlightenment is helping humanities scholars track the dispersion of great ideas across the world during that time, from Voltaire to Benjamin Franklin and everyone in between.

Relevant links:
https://www2.bc.edu/~jonescq/mb851/Mar12/PadgettAnsell_AJS_1993.pdf
http://republicofletters.stanford.edu/index.html

Great Social Networks in History

Survival analysis is all about studying how long until an event occurs--it's used in marketing to study how long a customer stays with a service, in epidemiology to estimate the duration of survival of a patient with some illness, and in social science to understand how the characteristics of a war inform how long the war goes on.  This episode talks about the special challenges associated with survival analysis, and the tools that (data) scientists use to answer all kinds of duration-related questions.

Survival Analysis

First up in this episode: a crash course in natural language processing, and important steps if you want to use machine learning techniques on text data.  Then we'll take that NLP know-how and talk about a really cool analysis of State of the Union text, which analyzes the topics and word choices of every President from Washington to Obama.

Relevant link:
https://civisanalytics.com/blog/data-science/2016/01/15/data-science-on-state-of-the-union-addresses/

Text Analysis on the State Of The Union

"People with a family history of heart disease are more likely to eat healthy foods, and have a high incidence of heart attacks."  Did the healthy food cause the heart attacks?  Probably not.  But establishing causal links is extremely tricky, and extremely important to get right if you're trying to help students, test new medicines, or just optimize a website.  In this episode, we'll unpack randomized experiments, like AB tests, and maybe you'll be smarter as a result.  Will you be smarter BECAUSE of this episode?  Well, tough to say for sure...

Relevant link:
http://tylervigen.com/spurious-correlations

Experiments and Messy, Tricky Causality

This episode started out as a discussion of recommendation engines, like Netflix uses to suggest movies.  There's still a lot of that in here.  But a related topic, which is both interesting and important, is how to keep data private in the era of large-scale recommendation engines--what mistakes have been made surrounding supposedly anonymized data, how data ends up de-anonymized, and why it matters for you.

Relevant links:
http://www.netflixprize.com/
http://bits.blogs.nytimes.com/2010/03/12/netflix-cancels-contest-plans-and-settles-suit/?_r=0
http://arxiv.org/PS_cache/cs/pdf/0610/0610105v2.pdf

Recommendation Engines and Privacy

Summary: For our inaugural podcast QCon chair Wesley Reisz talks to Adrian Cockcroft, who works for Battery Ventures where he advises the firm and its portfolio companies about technology issues and also assists with deal sourcing and due diligence.
Why listen to this podcast
•	Over the last year a large number of frameworks and libraries for building microservices have emerged and we're seeing a lot of rapid change.
•	The stack you choose will often be based on the main language you use, so for example Netflix’s stack is language agnostic but the tooling is very Java-centric.
•	Architectural choices can have a profound impact on success, so some-thing as simple as an overly long timeout with retries can cause your system to suffer from a congestion collapse problem.  Have a large timeout at the edge, and progressively smaller and smaller timeouts as you get deeper into the system.
•	Start with a monolith and move to a distributed architecture when you need to because of team size or to get better separation of concerns.
•	Other disruptive trends include “serverless architecture” like AWS Lambda, and open source itself which is perhaps the true definition of disruptive.
More on this
•	You can access our in-depth coverage on microservices http://bit.ly/1SRnFz3.
•	You can also subscribe to our newsletter to receive weekly updates on the hottest topics from professional software development. http://bit.ly/24x3IVq
•	Attend Adrian Cockcroft 's session at QCon New York 2016, Jun 13-17. http://bit.ly/1TvzlDQ 


Show Notes

Adrian Cockcroft on Microservices, Terraservices and Serverless Computing

Competing in a machine learning competition on Kaggle is a kind of rite of passage for data scientists.  Losing unexpectedly at the very end of the contest is also something that a lot of us have experienced.  It's not just bad luck: a very specific combination of overfitting on popular competitions can take someone who is in the top few spots in the final days of a contest and bump them down hundreds of slots in the final tally.

How to Lose at Kaggle

This week: everybody's favorite WWII-era classifier metric!  But it's not just for winning wars, it's a fantastic go-to metric for all your classifier quality needs.

Rock the ROC Curve

Wesley Reisz talks to Daniel Bryant on moving from monoliths to micro-services, covering bounded contexts, when to break up micro-services, event storming, practices like observability and tracing, and more.

Why listen to this podcast:

- Migrating a monolith to micro-services is best done by breaking off a valuable but not critical part first.
 - Designing a greenfield application as micro-services requires a strong understanding of the domain.
 - When a request enters the system, it needs to be tagged with a correlation id that flows down to all fan-out service requests.
 - Observability and metrics are essential parts to include when moving micro-services to production.
 - A service mesh allows you to scale services and permit binary transports without losing observability.

More on this: Quick scan our curated show notes on InfoQ http://bit.ly/2pFYBiT

You can also subscribe to the InfoQ newsletter to receive weekly updates on the hottest topics from professional software development. bit.ly/24x3IVq

Subscribe: www.youtube.com/infoq
Like InfoQ on Facebook: bit.ly/2jmlyG8
Follow on Twitter: twitter.com/InfoQ
Follow on LinkedIn: www.linkedin.com/company/infoq
Want to see extented shownotes? Check the landing page on InfoQ: http://bit.ly/2pFYBiT

InfoQ Podcast

Daniel Bryant on Microservices and Domain Driven Design

Wesley Reisz talks to Sid Anand, a data architect at cybersecurity company Agari, about building cloud-native data pipelines. The focus of their discussion is around a solution Agari uses that is built from Amazon Kinesis Streams, serverless functions, and auto scaling groups.

Sid Anand is an architect at Agari, and a former technical architect at eBay, Netflix, and LinkedIn.  He has 15 years of data infrastructure experience at scale, is a PMC for Apache Airflow, and is also a program committee chair for QCon San Francisco and QCon London.

Why listen to this podcast

 - Real-time data pipeline processing is very latency sensitive
 - Micro-batching allows much smaller amounts of data to be processed
 - Use the appropriate data store (or stores) to support the use of the dataIngesting data quickly into a clean database with minimal indexes can be fast
 - Communicate using a messaging system that supports schema evolution

More on this: Quick scan our curated show notes on InfoQ http://bit.ly/2rJU9nB

You can also subscribe to the InfoQ newsletter to receive weekly updates on the hottest topics from professional software development. bit.ly/24x3IVq

Subscribe: www.youtube.com/infoq
Like InfoQ on Facebook: bit.ly/2jmlyG8
Follow on Twitter: twitter.com/InfoQ
Follow on LinkedIn: www.linkedin.com/company/infoq
Want to see extented shownotes? Check the landing page on InfoQ: http://bit.ly/2rJU9nB

Sid Anand on Building Agari’s Cloud-native Data Pipelines with AWS Kinesis and Serverless

In this episode of the SuperDataScience Podcast, I chat with CEO & Co-founder at SFL Scientific Michael Segala. You will discuss about several exciting case studies from the consulting world, see how Data Science can be used in different areas of an organization, learn how it drives real value into business, and finally get valuable insights on data strategy and fantastic tips for those who are building a career in Data Science. 

If you enjoyed this episode, check out show notes, resources, and more at http://www.superdatascience.com/65

The Super Data Science Podcast: Big Data | Analytics Careers | Mentors | Success

SDS 065: How Data Science Brings Value through Consulting Firms

We're back again with friend of the pod Walt, former software engineer extraordinaire and current data scientist extraordinaire, to talk about some best practices from software engineering that are ready to jump the fence over to data science.  If last week's episode was for software engineers who are interested in becoming more like data scientists, then this week's episode is for data scientists who are looking to improve their game with best practices from software engineering.

What Data Scientists Can Learn from Software Engineers

This week's episode is just in time for JupyterCon in NYC, August 22-25...

Jupyter notebooks are probably familiar to a lot of data nerds out there as a great open-source tool for exploring data, doing quick visualizations, and packaging code snippets with explanations for sharing your work with others.  If you're not a data person, or you are but you haven't tried out Jupyter notebooks yet, here's your nudge to go give them a try.  In this episode we'll go back to the old days, before notebooks, and talk about all the ways that data scientists like to work that wasn't particularly well-suited to the command line + text editor setup, and talk about how notebooks have evolved over their lifetime to become even more powerful and well-suited to the data scientist's workflow.

Jupyter Notebooks

Even as we rely more and more on machine learning algorithms to help with everyday decision-making, we're learning more and more about how they're frighteningly easy to fool sometimes.  Today we have a roundup of a few successful efforts to create robust adversarial examples, including what it means for an adversarial example to be robust and what this might mean for machine learning in the future.

Adversarial Examples

Nora Jones, a senior software engineer on Netflix’ Chaos Team, talks with Wesley Reisz about what Chaos Engineering means today. She covers what it takes to build a practice, how to establish a strategy, defines cost of impact, and covers key technical considerations when leveraging chaos engineering.

Why listen to this podcast:

- Chaos engineering is a discipline where you formulate hypotheses, perform experiments, and evaluate the results afterwards.
- Injecting a bit of failure over time is going to make your system more resilient in the end.
- Start with Tier 2 or non-critical services first, and build up success stories to grow chaos further.
- As systems become more and more distributed, there becomes a higher need for chaos engineering.
- If you’re running your first experiment, get your service owners in a war room and get them to monitor the results of the test as it is running.

More on this: Quick scan our curated show notes on InfoQ http://bit.ly/2vJoimw

You can also subscribe to the InfoQ newsletter to receive weekly updates on the hottest topics from professional software development. bit.ly/24x3IVq

Subscribe: www.youtube.com/infoq
Like InfoQ on Facebook: bit.ly/2jmlyG8
Follow on Twitter: twitter.com/InfoQ
Follow on LinkedIn: www.linkedin.com/company/infoq
Want to see extended shownotes? Check the landing page on InfoQ: http://bit.ly/2vJoimw

Nora Jones on Establishing, Growing, and Maturing a Chaos Engineering Practice

It's been a busy hurricane season in the Southeastern United States, with millions of people making life-or-death decisions based on the forecasts around where the hurricanes will hit and with what intensity.  In this episode we'll deconstruct those models, talking about the different types of models, the theory behind them, and how they've evolved through the years.

Hurricane Forecasting

As data science matures as a field, it's becoming clearer what attributes a data science team needs to have to elevate their work to the next level.  Most of our episodes are about the cool work being done by other people, but this one summarizes some thinking Katie's been doing herself around how to guide data science teams toward more mature, effective practices.  We'll go through five key characteristics of great data science teams, which we collectively refer to as "disciplined data science," and why they matter.

Disciplined Data Science

It's quite common for survey respondents not to be representative of the larger population from which they are drawn.  But if you're a researcher, you need to study the larger population using data from your survey respondents, so what should you do?  Reweighting the survey data, so that things like demographic distributions look similar between the survey and general populations, is a standard technique and in this episode we'll talk about survey raking, a way to calculate survey weights when there are several distributions of interest that need to be matched.

Survey Raking

This week, we've got a fun paper by our friends at Google about the hidden costs of maintaining machine learning workflows.  If you've worked in software before, you're probably familiar with the idea of technical debt, which are inefficiencies that crop up in the code when you're trying to go fast.  You take shortcuts, hard-code variable values, skimp on the documentation, and generally write not-that-great code in order to get something done quickly, and then end up paying for it later on.  This is technical debt, and it's particularly easy to accrue with machine learning workflows.  That's the premise of this episode's paper.

Machine Learning: The High Interest Credit Card of Technical Debt

In this podcast Shane Hastie, Lead Editor for Culture & Methods, spoke to Josh Evans, former engineering manager at Netflix on how Netflix does DevOps and the freedom and responsibility culture that undermines their way of working. 

Why listen to this podcast:

•	There are many interpretations of the term DevOps, it is a useful shorthand for a wide variety of technologies and approaches 
•	“You build it, you run it” is the concrete application of the freedom and responsibility culture
•	When building a platform tool make it so easy to use that the product teams are not tempted to try and build something for themselves
•	Product teams are free to experiment and learn, which can feel chaotic and is a valuable part of the freedom and responsibility culture
•	The value of blameless and safe incident reviews – the goal is to learn and find patterns and use that information to present whole classes of failure from happening in the future
•	Don’t view the value stream in a fragmented way – see the whole end to end system with all its interactions and dependencies and optimize the system as a cohesive whole rather than different tools and domains

More on this: Quick scan our curated show notes on InfoQ http://bit.ly/2mtCIr1

You can also subscribe to the InfoQ newsletter to receive weekly updates on the hottest topics from professional software development. bit.ly/24x3IVq

Subscribe: www.youtube.com/infoq
Like InfoQ on Facebook: bit.ly/2jmlyG8
Follow on Twitter: twitter.com/InfoQ
Follow on LinkedIn: www.linkedin.com/company/infoq
Want to see extented shownotes? Check the landing page on InfoQ: http://bit.ly/2mtCIr1

Josh Evans on DevOps at Netflix

Picking up after last week's episode about maximal margin classifiers, this week we'll go into the kernel trick and how that (combined with maximal margin algorithms) gives us the much-vaunted support vector machine.

The Kernel Trick and Support Vector Machines

The Fourier transform is one of the handiest tools in signal processing for dealing with periodic time series data. Using a Fourier transform, you can break apart a complex periodic function into a bunch of sine and cosine waves, and figure out what the amplitude, frequency and offset of those component waves are. It's a really handy way of re-expressing periodic data--you'll never look at a time series graph the same way again.

The Fourier Transform

In this episode of the Data Show, I spoke with Fabian Yamaguchi, chief scientist at ShiftLeft. His 2015 Ph.D. dissertation sketched out how the combination of static analysis, graph mining, and machine learning, can be used to develop tools to augment security analysts. In a recent post, I argued for machine learning tools to augment teams responsible for deploying and managing models in production (machine learning engineers). These are part of a general trend of using machine learning to develop and manage the software systems of tomorrow. Yamaguchi’s work is step one in this direction: using machine learning to reduce the number of security vulnerabilities in complex software products.

How Machine Learning Can Be Used to Write More Secure Computer Programs

One interesting trend we've noted recently is the proliferation of papers, articles and blog posts about data science that don't just tell the result--they include data and code that allow anyone to repeat the analysis. It's far from universal (for a timely counterpoint, read this article <http://www.sciencemag.org/news/2018/02/missing-data-hinder-replication-artificial-intelligence-studies>), but we seem to be moving toward a new normal where data science conclusions are expected to be shown, not just told.

Relevant links:
https://github.com/fivethirtyeight/data
https://blog.patricktriest.com/police-data-python/

Open Data and Open Science

A few weeks ago, we podcasted about a neural network that was being touted as "better than doctors" in diagnosing pneumonia from chest x-rays, and how the underlying dataset used to train the algorithm raised some serious questions. We're back again this week with further developments, as the author of the original blog post pointed us toward more developments. All in all, there's a lot more clarity now around how the authors arrived at their original "better than doctors" claim, and a number of adjustments and improvements as the original result was de/re-constructed. 

Anyway, there are a few things that are cool about this. First, it's a worthwhile follow-up to a popular recent episode. Second, it goes *inside* an analysis to see what things like imbalanced classes, outliers, and (possible) signal leakage can do to real science. And last, it raises a really interesting question in an age when computers are often claimed to be better than humans: what do those claims really mean?

Relevant links:
https://lukeoakdenrayner.wordpress.com/2018/01/24/chexnet-an-in-depth-review/

What makes a machine learning algorithm "superhuman"?

After all the back-patting around making data science datasets and code more openly available, we figured it was time to also dump a bucket of cold water on everyone's heads and talk about the things that can go wrong when data and code is a little too open. 

In this episode, we'll talk about two interesting recent examples: a de-identified medical dataset in Australia that was re-identified so specific celebrities and athletes could be matched to their medical records, and a series of military bases that were spotted in a public fitness tracker dataset.

When Private Data Isn't Private Anymore

This week's episodes is for data scientists, sure, but also for data science managers and executives at companies with data science teams. These folks all think very differently about the same question: what should a data science team be working on? And how should that decision be made? That's the subject of a talk that I (Katie) gave at Strata Data in early March, about how my co-department head and I select projects for our team to work on.

We have several goals in data science project selection at Civis Analytics (where I work), which can be summarized under "balance the best attributes of bottom-up and top-down decision-making." We achieve this balance, or at least get pretty close, using a process we've come to call the Idea Factory (after a great book about Bell Labs). This talk is about that process, how it works in the real world of a data science company and how we see it working in the data science programs of other companies.

Relevant links: https://conferences.oreilly.com/strata/strata-ca/public/schedule/detail/63905

How to pick projects for a professional data science team

If you've done image recognition or computer vision tasks with a neural network, you've probably used a convolutional neural net. This episode is all about the architecture and implementation details of convolutional networks, and the tricks that make them so good at image tasks.

Convolutional Neural Nets

In this episode of the AI in Industry podcast, we speak with Marshall Choy, VP of Product at SambaNova, an AI hardware firm based in the Bay Area. SambaNova was founded by a number of Oracle and Sun Micro Systems alumni. We speak with Choy on two fundamental questions:  How will business models fundamentally change with respect to new AI hardware capabilities? How can business leaders think about their AI hardware needs?  SambaNova is one of many firms that's going to be advertising at the Kisaco Research AI Hardware Summit in Beijing June 4th and 5th.

How Business Leaders Should Think About AI Hardware

Open source software is ubiquitous throughout data science, and enables the work of nearly every data scientist in some way or another. Open source projects, however, are disproportionately maintained by a small number of individuals, some of whom are institutionally supported, but many of whom do this maintenance on a purely volunteer basis. The health of the data science ecosystem depends on the support of open source projects, on an individual and institutional level.

https://hdsr.mitpress.mit.edu/pub/xsrt4zs2/release/2

The Data Science Open Source Ecosystem

This is a re-release of an episode that originally ran in October 2019.

If you’re trying to manage a project that serves up analytics data for a few very distinct uses, you’d be wise to consider having custom solutions for each use case that are optimized for the needs and constraints of that use cases. You also wouldn’t be YouTube, which found themselves with this problem (gigantic data needs and several very different use cases of what they needed to do with that data) and went a different way: they built one analytics data system to serve them all. Procella, the system they built, is the topic of our episode today: by deconstructing the system, we dig into the four motivating uses of this system, the complexity they had to introduce to service all four uses simultaneously, and the impressive engineering that has to go into building something that “just works.”

Procella: YouTube's super-system for analytics data storage

A few weeks ago, we put out a call for data scientists interested in issues of race and racism, or people studying how those topics can be studied with data science methods, should get in touch to come talk to our audience about their work. This week we’re excited to bring on Todd Hendricks, Bay Area data scientist and a volunteer who reached out to tell us about his studies with the Stanford Open Policing dataset.

A Data Science Take on Open Policing Data

The data science and artificial intelligence community has made amazing strides in the past few years to algorithmically automate portions of the healthcare process. This episode looks at two computer vision algorithms, one that diagnoses diabetic retinopathy and another that classifies liver cancer, and asks the question—are patients now getting better care, and achieving better outcomes, with these algorithms in the mix? The answer isn’t no, exactly, but it’s not a resounding yes, because these algorithms interact with a very complex system (the healthcare system) and other shortcomings of that system are proving hard to automate away. Getting a faster diagnosis from an image might not be an improvement if the image is now harder to capture (because of strict data quality requirements associated with the algorithm that wouldn’t stop a human doing the same job). Likewise, an algorithm getting a prediction mostly correct might not be an overall benefit if it introduces more dramatic failures when the prediction happens to be wrong. For every data scientist whose work is deployed into some kind of product, and is being used to solve real-world problems, these papers underscore how important and difficult it is to consider all the context around those problems.

A Reality Check on AI-Driven Medical Assistants

All good things must come to an end, including this podcast. This is the last episode we plan to release, and it doesn’t cover data science—it’s mostly reminiscing, thanking our wonderful audience (that’s you!), and marveling at how this thing that started out as a side project grew into a huge part of our lives for over 5 years.

It’s been a ride, and a real pleasure and privilege to talk to you each week. Thanks, best wishes, and good night!

—Katie and Ben