Kafka and Event Sourcing

Very nice talk on using Kafka as an event store in a distributed architecture.

Reducing Microservice Complexity with Kafka and Reactive Streams - by Jim Riecken

• start from a monolith
  • single build pipeline
  • good for small teams
  • doesn’t scale well
• strangler pattern
  • tease out small services
  • monolith becomes a facade that calls into microservices
• microservices
  • clear ownership
  • fast build times
  • independently scalable
  • allows innovation and new technology
• Cons
  • latency
  • cascading failure
  • uptime is based on combined critical service path
• non-essential calls should be asynchronous
  • decoupling
  • producers don’t need to be aware of consumers
  • define delivery requirements
  • buffering for slow consumers
• kafka
  • append only
    • fast O(1)
    • LinkedIn sent 800billion/day (2015)
  • broker data persisted to disk
  • topics + partitions
    • balanced or hashed
  • persistence of hours to weeks
  • pulling consumers
    • save current index
  • immutable log files
  • consumers can pick up where they left off due to upgrades or downtime