From Transactions to Streams

Martin Kleppmann explores using event streams and Kafka for keeping data in sync across heterogeneous systems, and compares this approach to distributed transactions, discussing what consistency guarantees can it offer, and how it fares in the face of failure.

The Hardest Part About Microservices is your Data

Of the reasons we attempt a microservices architecture, chief among them is allowing your teams to […] be autonomous, capable of making decisions about how to best implement and operate their services, and free to make changes as quickly as the business may desire.

To gain this autonomy, […] don’t share a single database across services because then you run into conflicts like competing read/write patterns, data-model conflicts, coordination challenges, etc. But a single database does afford us a lot of safeties and conveniences: ACID transactions, single place to look, well understood (kinda?), one place to manage, etc. So when building microservices how do we reconcile these safeties with splitting up our database into multiple smaller databases?

Kafka and Event Sourcing

Very nice talk on using Kafka as an event store in a distributed architecture.

Reducing Microservice Complexity with Kafka and Reactive Streams - by Jim Riecken

  • start from a monolith
    • single build pipeline
    • good for small teams
    • doesn’t scale well
  • strangler pattern
    • tease out small services
    • monolith becomes a facade that calls into microservices
  • microservices
    • clear ownership
    • fast build times
    • independently scalable
    • allows innovation and new technology
  • Cons
    • latency
    • cascading failure
    • uptime is based on combined critical service path
  • non-essential calls should be asynchronous
    • decoupling
    • producers don’t need to be aware of consumers
    • define delivery requirements
    • buffering for slow consumers
  • kafka
    • append only
      • fast O(1)
      • LinkedIn sent 800billion/day (2015)
    • broker data persisted to disk
    • topics + partitions
      • balanced or hashed
    • persistence of hours to weeks
    • pulling consumers
      • save current index
    • immutable log files
    • consumers can pick up where they left off due to upgrades or downtime

Service Fabric Common Questions

In round 8 of the Service Fabric Q&A, they announced a “common questions” page to hopefully answer the frequently asked questions they’ve gotten almost every time.

Update: Another gem I became aware of in this session is Microsoft.Diagnostic.EventFlow which allows you to log to Microsoft.Extensions.Logging and have it persist to Elasticsearch (and the ELK stack more specifically.)

Service Fabric Windows Container

I’ve been playing with Azure Service Fabric and testing out the latest preview with Windows container support. My goal is to deploy a stateless OWIN self-hosted Web API in a container to service fabric. I saw a cool demo of an application upgrade, so I stole the idea of an API that simply returns a color (blue.) Version 2 of the service returns a different color (red), so you can actually watch the rolling upgrade across the cluster as the color changes from blue to red in several load-balanced clients.

rolling upgrade


using Microsoft.Owin.Hosting;
using System;
using System.Threading;

namespace ColorPicker
    class Program
        static void Main(string[] args)
            string baseAddress = "http://*:80/";

            using (WebApp.Start<Startup>(url: baseAddress))
                Console.WriteLine($"ColorPicker listening at {baseAddress}");


using Owin;
using System.Web.Http;

namespace ColorPicker
    public class Startup
        public void Configuration(IAppBuilder app)
            HttpConfiguration config = new HttpConfiguration();
                name: "DefaultApi",
                routeTemplate: "api/{controller}/{id}",
                defaults: new { id = RouteParameter.Optional }


using System.Web.Http;

namespace ColorPicker
    public class ColorController : ApiController
        public string Get()
            return "#0000FF";


FROM microsoft/windowsservercore
ADD . /app
ENTRYPOINT ["cmd.exe", "/k", "ColorPicker.exe"]

Docker Registry

The service manifest only lets you specify the name of the container, which means you have to deploy it to a registry. I don’t want to push anything to DockerHub, so instead I decided to give Docker Registry a try.

I’m using a macbook, so I followed the quick guide to run the registry as a container.

PS C:\> docker run -d -p 5000:5000 --name registry registry:2

I’m running Windows in a VM and tried pushing my Windows docker image to the registry running on the mac host.

PS C:\> docker build -t colorpicker .
PS C:\> docker tag colorpicker
PS C:\> docker push

This surprisingly worked, but gave me a TLS error:

The push refers to a repository []
Get http: server gave HTTP response to HTTPS client

Make it insecure

It turns out the docker client requires TLS by default, but the docs also explain how to run it insecurely for testing. To disable security, you have to edit the docker config file on the client, and whitelist the insecure registry you want to connect to.

The instructions tell you to edit the docker file directly, which only applies to *nix clients. I dug a little deeper into how to configure this for both Docker for Mac and Windows.

Docker for Mac

Docker for Mac provisions a HyperKit VM based on Alpine Linux. They do some interesting stuff with git to modify files in the docker app (such as a daemon.json file,) and then reload the VM. However, they have since added an option in the advanced settings dialog to add insecure registries.

mac settings

If you still wish to connect to the host (for debugging, or other reasons,) you have to use screen rather than ssh. Found that little nugget on this blog. Any changes to the host will be lost whenever the daemon is restarted.

Press enter to get a prompt. Disconnect with ctrl+a d.

$ screen ~/Library/Containers/com.docker.docker/Data/com.docker.driver.amd64-linux/tty

Docker for Windows

Docker for Windows lets you run Linux containers on Windows 10 by provisioning a minimal Linux VM using Hyper-V. I’m not aware of any way to access the host environment, and the stable version of docker does not expose any settings to configure the docker file.

But I want to push my windows container, not a linux container. Why would I need Docker for Windows? As luck would have it, I needed to install the beta release in order to get the option to switch between linux and windows containers. The beta settings dialog includes daemon options which lets you set insecure registries. This is not currently available in the stable release.

windows settings

Another point worth mentioning is that the docker daemon would not start on my Windows VM when set to use linux containers, which has something to do with nested virtualization (running a VM inside a VM.) Since I don’t really care about linux containers at the moment, I simply switched to windows containers, and slowly backed into the bushes.

homer scare

Let’s try again…

Now that our client is configured to trust our insecure registry, let’s try the push again.

PS C:\> docker push
The push refers to a repository []
242b7a69a21f: Pushed
0281ebf270fd: Pushed
4777122753b8: Pushed
de57d9086f9a: Skipped foreign layer
f358be10862c: Skipped foreign layer
latest: digest: sha256:7d3ff34231abec035813d36f1cfe48c737e15ba9b6db9dd238e5c1f166bcb73d size: 1570