The modern world runs on Data. In this talk we will cover how Gophers of any level can easily build Data Pipelines in Go with Kafka and Cassandra. At the end, we will look at how GE has written a Data Pipeline in Go that can handle over 800,000 writes per second of industrial time series data.

Introduction to Data Pipelines:
- What are data pipelines
- Why Go is a good language for them

Package structure:
- How to lay out our data pipeline’s packages
- How data will flow throughout the application
- Example code

Writing integration tests:
- Using docker to write integration tests
- Simulating downtime using docker pause on Kafka or Cassandra
- Example code

Data source: Kafka:
- What is Kafka and how we can use it with Go
- How to ensure no data loss is possible with good offset management
- Reading from multiple Kafka partitions with a high level consumer
- Example code

Performing ETL on the data and business logic:
- Parsing data
- Data ETL
- Performing intermediary business logic
- Example code

Persistent Data Storage: Cassandra:
- Setting up gocql to write to Cassandra
- Best practices for writing to Cassandra
- Example code

Demo: Processing hundreds of thousands of message:
- Finished version of our demo application
- Running in a Kubernetes cluster at scale
- Killing components, seeing how it recovers
- Finished example code

Example use case: Go Data Pipeline at GE Digital:
- Example data pipeline that’s running in production at GE Digital
- Production results of a similar data pipeline with over 800,000 writes per second