A practical introduction to data engineering on the powerful Snowflake cloud data platform.
Data engineers create the pipelines that ingest raw data, transform it, and funnel it to the analysts and professionals who need it. The Snowflake cloud data platform provides a suite of productivity-focused tools and features that simplify building and maintaining data pipelines. In Snowflake Data Engineering, Snowflake Data Superhero Maja Ferle shows you how to get started.
In Snowflake Data Engineering you will learn how to:
• Ingest data into Snowflake from both cloud and local file systems
• Transform data using functions, stored procedures, and SQL
• Orchestrate data pipelines with streams and tasks, and monitor their execution
• Use Snowpark to run Python code in your pipelines
• Deploy Snowflake objects and code using continuous integration principles
• Optimize performance and costs when ingesting data into Snowflake
Snowflake Data Engineering reveals how Snowflake makes it easy to work with unstructured data, set up continuous ingestion with Snowpipe, and keep your data safe and secure with best-in-class data governance features. Along the way, you’ll practice the most important data engineering tasks as you work through relevant hands-on examples. Throughout, author Maja Ferle shares design tips drawn from her years of experience to ensure your pipeline follows the best practices of software engineering, security, and data governance.
Foreword by Joe Reis.
Purchase of the print book includes a free eBook in PDF and ePub formats from Manning Publications.
About the technology
Pipelines that ingest and transform raw data are the lifeblood of business analytics, and data engineers rely on Snowflake to help them deliver those pipelines efficiently. Snowflake is a full-service cloud-based platform that handles everything from near-infinite storage, fast elastic compute services, inbuilt AI/ML capabilities like vector search, text-to-SQL, code generation, and more. This book gives you what you need to create effective data pipelines on the Snowflake platform.
About the book
Snowflake Data Engineering guides you skill-by-skill through accomplishing on-the-job data engineering tasks using Snowflake. You’ll start by building your first simple pipeline and then expand it by adding increasingly powerful features, including data governance and security, adding CI/CD into your pipelines, and even augmenting data with generative AI. You’ll be amazed how far you can go in just a few short chapters!
What's inside
• Ingest data from the cloud, APIs, or Snowflake Marketplace
• Orchestrate data pipelines with streams and tasks
• Optimize performance and cost
About the reader
For software developers and data analysts. Readers should know the basics of SQL and the Cloud.
About the author
Maja Ferle is a Snowflake Subject Matter Expert and a Snowflake Data Superhero who holds the SnowPro Advanced Data Engineer and the SnowPro Advanced Data Analyst certifications.
Table of Contents
Part 1
1 Data engineering with Snowflake
2 Creating your first data pipeline
Part 2
3 Best practices for data staging
4 Transforming data
5 Continuous data ingestion
6 Executing code natively with Snowpark
7 Augmenting data with outputs from large language models
8 Optimizing query performance
9 Controlling costs
10 Data governance and access control
Part 3
11 Designing data pipelines
12 Ingesting data incrementally
13 Orchestrating data pipelines
14 Testing for data integrity and completeness
15 Data pipeline continuous integration
About the Author
Maja Ferle is a seasoned data architect with more than 30 years of experience in data analytics, data warehousing, business intelligence, data engineering, data modeling, and database administration. She holds the SnowPro Advanced Data Engineer and the SnowPro Advanced Data Analyst certifications. She is also a Snowflake Subject Matter Expert and a Snowflake Data Superhero.
Ebook License
End-User Warranty And License Agreement
1. Grant Of License
Manning Has Authorized The Download By You Of An Unrestricted Number Of Copies Of The Electronic Book (Ebook) In Any Of The Available Formats. Manning Grants You A Nonexclusive, Nontransferable License To Use The Ebook According To The Terms And Conditions Herein. This License Agreement Permits You To Install The Ebook On Any And All Your Devices For Your Personal Use Only.
2. Restrictions
You Shall Not: (1) Share, Resell, Rent, Assign, Timeshare, Distribute, Or Transfer All Or Part Of The Ebook Or Any Rights Granted Hereunder To Any Other Person; (2) Duplicate The Ebook, Except For A Single Backup Or Archival Copy; (3) Remove Any Proprietary Notices, Labels, Or Marks From The Ebook; (4) Transfer Or Sublicense Title To The Ebook To Any Other Party.
3. Intellectual Property Protection
The Ebook Is Owned By Manning And Is Protected By United States And International Copyright And Other Intellectual Property Laws. Manning Reserves All Rights In The Ebook Not Expressly Granted Herein. This License And Your Right To Use The Ebook Terminate Automatically If You Violate Any Part Of This Agreement. In The Event Of Termination, You Must Remove The Original And Any Copies Of The Ebook From All Your Devices.
4. Source Code Supplementary Material
Any Source Code Files Provided As A Supplement To The Book Are Freely Available To The Public For Download. Reuse Of The Code Is Permitted, In Whole Or In Part, Including The Creation Of Derivative Works, Provided That You Acknowledge That You Are Using It And Identify The Source: Title, Publisher And Year.
5. Limited Warranty
Manning Warrants That The Ebook Files, A Copy Of Which You Are Authorized To Download, Are Free From Defects In The Operational Sense That They Can Be Read By A Pdf Reader Or Epub Reader, Or Other. Except For This Express Limited Warranty, Manning Makes And You Receive No Warranties, Express, Implied, Statutory Or In Any Communication With You, And Manning Specifically Disclaims Any Other Warranty Including The Implied Warranty Of Merchantability Or Fitness Or A Particular Purpose. Manning Does Not Warrant That The Operation Of The Ebook Will Be Uninterrupted Or Error Free. If The Ebook Was Purchased In The United States, The Above Exclusions May Not Apply To You As Some States Do Not Allow The Exclusion Of Implied Warranties. In Addition To The Above Warranty Rights, You May Also Have Other Rights That Vary From State To State.
6. Limitation Of Liability
In No Event Will Manning Be Liable For Any Damages, Whether Arising For Tort Or Contract, Including Loss Of Data, Lost Profits, Or Other Special, Incidental, Consequential, Or Indirect Damages Arising Out Of The Use Or Inability To Use The Ebook.
7. General
This Agreement Constitutes The Entire Agreement Between You And Manning And Supersedes Any Prior Agreement Concerning The Ebook. This Agreement Is Governed By The Laws Of The State Of New York.
