Human-in-the-Loop Machine Learning lays out methods for humans and machines to work together effectively.
Summary
Most machine learning systems that are deployed in the world today learn from human feedback. However, most machine learning courses focus almost exclusively on the algorithms, not the human-computer interaction part of the systems. This can leave a big knowledge gap for data scientists working in real-world machine learning, where data scientists spend more time on data management than on building algorithms. Human-in-the-Loop Machine Learning is a practical guide to optimizing the entire machine learning process, including techniques for annotation, active learning, transfer learning, and using machine learning to optimize every step of the process.
Purchase of the print book includes a free eBook in PDF, Kindle, and ePub formats from Manning Publications.
About the technology
Machine learning applications perform better with human feedback. Keeping the right people in the loop improves the accuracy of models, reduces errors in data, lowers costs, and helps you ship models faster.
About the book
Human-in-the-Loop Machine Learning lays out methods for humans and machines to work together effectively. You’ll find best practices on selecting sample data for human feedback, quality control for human annotations, and designing annotation interfaces. You’ll learn to create training data for labeling, object detection, and semantic segmentation, sequence labeling, and more. The book starts with the basics and progresses to advanced techniques like transfer learning and self-supervision within annotation workflows.
What's inside
Identifying the right training and evaluation data
Finding and managing people to annotate data
Selecting annotation quality control strategies
Designing interfaces to improve accuracy and efficiency
About the author
Robert (Munro) Monarch is a data scientist and engineer who has built machine learning data for companies such as Apple, Amazon, Google, and IBM. He holds a PhD from Stanford.
Robert holds a PhD from Stanford focused on Human-in-the-Loop machine learning for healthcare and disaster response, and is a disaster response professional in addition to being a machine learning professional. A worked example throughout this text is classifying disaster-related messages from real disasters that Robert has helped respond to in the past.
Table of Contents
PART 1 - FIRST STEPS
1 Introduction to human-in-the-loop machine learning
2 Getting started with human-in-the-loop machine learning
PART 2 - ACTIVE LEARNING
3 Uncertainty sampling
4 Diversity sampling
5 Advanced active learning
6 Applying active learning to different machine learning tasks
PART 3 - ANNOTATION
7 Working with the people annotating your data
8 Quality control for data annotation
9 Advanced data annotation and augmentation
10 Annotation quality for different machine learning tasks
PART 4 - HUMAN–COMPUTER INTERACTION FOR MACHINE LEARNING
11 Interfaces for data annotation
12 Human-in-the-loop machine learning products
eBook License
End-User Warranty and License Agreement
1. Grant of License
Manning has authorized the download by you of an unrestricted number of copies of the electronic book (ebook) in any of the available formats. Manning grants you a nonexclusive, nontransferable license to use the ebook according to the terms and conditions herein. This License Agreement permits you to install the ebook on any and all your devices for your personal use only.
2. Restrictions
You shall not: (1) share, resell, rent, assign, timeshare, distribute, or transfer all or part of the ebook or any rights granted hereunder to any other person; (2) duplicate the ebook, except for a single backup or archival copy; (3) remove any proprietary notices, labels, or marks from the ebook; (4) transfer or sublicense title to the ebook to any other party.
3. Intellectual Property Protection
The ebook is owned by Manning and is protected by United States and international copyright and other intellectual property laws. Manning reserves all rights in the ebook not expressly granted herein. This license and your right to use the ebook terminate automatically if you violate any part of this Agreement. In the event of termination, you must remove the original and any copies of the ebook from all your devices.
4. Source Code Supplementary Material
Any source code files provided as a supplement to the book are freely available to the public for download. Reuse of the code is permitted, in whole or in part, including the creation of derivative works, provided that you acknowledge that you are using it and identify the source: title, publisher and year.
5. Limited Warranty
Manning warrants that the ebook files, a copy of which you are authorized to download, are free from defects in the operational sense that they can be read by a PDF Reader or ePub reader, or other. EXCEPT FOR THIS EXPRESS LIMITED WARRANTY, MANNING MAKES AND YOU RECEIVE NO WARRANTIES, EXPRESS, IMPLIED, STATUTORY OR IN ANY COMMUNICATION WITH YOU, AND MANNING SPECIFICALLY DISCLAIMS ANY OTHER WARRANTY INCLUDING THE IMPLIED WARRANTY OF MERCHANTABILITY OR FITNESS OR A PARTICULAR PURPOSE. MANNING DOES NOT WARRANT THAT THE OPERATION OF THE EBOOK WILL BE UNINTERRUPTED OR ERROR FREE. If the ebook was purchased in the United States, the above exclusions may not apply to you as some states do not allow the exclusion of implied warranties. In addition to the above warranty rights, you may also have other rights that vary from state to state.
6. Limitation of Liability
IN NO EVENT WILL MANNING BE LIABLE FOR ANY DAMAGES, WHETHER ARISING FOR TORT OR CONTRACT, INCLUDING LOSS OF DATA, LOST PROFITS, OR OTHER SPECIAL, INCIDENTAL, CONSEQUENTIAL, OR INDIRECT DAMAGES ARISING OUT OF THE USE OR INABILITY TO USE THE EBOOK.
7. General
This Agreement constitutes the entire agreement between you and Manning and supersedes any prior agreement concerning the ebook. This Agreement is governed by the laws of the State of New York.
