Martin Kleppmann explores using event streams and Kafka for keeping data in sync across heterogeneous systems, and compares this approach to distributed transactions, discussing what consistency guarantees can it offer, and how it fares in the face of failure.
Of the reasons we attempt a microservices architecture, chief among them is allowing your teams to […] be autonomous, capable of making decisions about how to best implement and operate their services, and free to make changes as quickly as the business may desire.
To gain this autonomy, […] don’t share a single database across services because then you run into conflicts like competing read/write patterns, data-model conflicts, coordination challenges, etc. But a single database does afford us a lot of safeties and conveniences: ACID transactions, single place to look, well understood (kinda?), one place to manage, etc. So when building microservices how do we reconcile these safeties with splitting up our database into multiple smaller databases?
http://blog.christianposta.com/microservices/the-hardest-part-about-microservices-data/
Very nice talk on using Kafka as an event store in a distributed architecture.
Reducing Microservice Complexity with Kafka and Reactive Streams - by Jim Riecken
In round 8 of the Service Fabric Q&A, they announced a “common questions” page to hopefully answer the frequently asked questions they’ve gotten almost every time.
https://docs.microsoft.com/en-us/azure/service-fabric/service-fabric-common-questions
Update: Another gem I became aware of in this session is Microsoft.Diagnostic.EventFlow which allows you to log to Microsoft.Extensions.Logging and have it persist to Elasticsearch (and the ELK stack more specifically.)
I’ve been playing with Azure Service Fabric and testing out the latest preview with Windows container support. My goal is to deploy a stateless OWIN self-hosted Web API in a container to service fabric. I saw a cool demo of an application upgrade, so I stole the idea of an API that simply returns a color (blue.) Version 2 of the service returns a different color (red), so you can actually watch the rolling upgrade across the cluster as the color changes from blue to red in several load-balanced clients.
Program.cs
using Microsoft.Owin.Hosting;
using System;
using System.Threading;
namespace ColorPicker
{
class Program
{
static void Main(string[] args)
{
string baseAddress = "http://*:80/";
using (WebApp.Start<Startup>(url: baseAddress))
{
Console.WriteLine($"ColorPicker listening at {baseAddress}");
Thread.Sleep(Timeout.Infinite);
}
}
}
}
Startup.cs
using Owin;
using System.Web.Http;
namespace ColorPicker
{
public class Startup
{
public void Configuration(IAppBuilder app)
{
HttpConfiguration config = new HttpConfiguration();
config.Routes.MapHttpRoute(
name: "DefaultApi",
routeTemplate: "api/{controller}/{id}",
defaults: new { id = RouteParameter.Optional }
);
config.Formatters.Remove(config.Formatters.XmlFormatter);
app.UseWebApi(config);
}
}
}
ColorController.cs
using System.Web.Http;
namespace ColorPicker
{
public class ColorController : ApiController
{
public string Get()
{
return "#0000FF";
}
}
}
Dockerfile
FROM microsoft/windowsservercore
ADD . /app
WORKDIR /app
ENTRYPOINT ["cmd.exe", "/k", "ColorPicker.exe"]
The service manifest only lets you specify the name of the container, which means you have to deploy it to a registry. I don’t want to push anything to DockerHub, so instead I decided to give Docker Registry a try.
I’m using a macbook, so I followed the quick guide to run the registry as a container.
PS C:\> docker run -d -p 5000:5000 --name registry registry:2
I’m running Windows in a VM and tried pushing my Windows docker image to the registry running on the mac host.
PS C:\> docker build -t colorpicker .
PS C:\> docker tag colorpicker 10.211.55.2:5000/colorpicker
PS C:\> docker push 10.211.55.2:5000/colorpicker
This surprisingly worked, but gave me a TLS error:
The push refers to a repository [10.211.55.2:5000/colorpicker]
Get https://10.211.55.2:5000/v2/: http: server gave HTTP response to HTTPS client
It turns out the docker client requires TLS by default, but the docs also explain how to run it insecurely for testing. To disable security, you have to edit the docker config file on the client, and whitelist the insecure registry you want to connect to.
The instructions tell you to edit the docker file directly, which only applies to *nix clients. I dug a little deeper into how to configure this for both Docker for Mac and Windows.
Docker for Mac provisions a HyperKit VM based on Alpine Linux. They do some interesting stuff with git to modify files in the docker app (such as a daemon.json file,) and then reload the VM. However, they have since added an option in the advanced settings dialog to add insecure registries.
If you still wish to connect to the host (for debugging, or other reasons,) you have to use screen rather than ssh. Found that little nugget on this blog. Any changes to the host will be lost whenever the daemon is restarted.
Press enter to get a prompt. Disconnect with ctrl+a d.
$ screen ~/Library/Containers/com.docker.docker/Data/com.docker.driver.amd64-linux/tty
Docker for Windows lets you run Linux containers on Windows 10 by provisioning a minimal Linux VM using Hyper-V. I’m not aware of any way to access the host environment, and the stable version of docker does not expose any settings to configure the docker file.
But I want to push my windows container, not a linux container. Why would I need Docker for Windows? As luck would have it, I needed to install the beta release in order to get the option to switch between linux and windows containers. The beta settings dialog includes daemon options which lets you set insecure registries. This is not currently available in the stable release.
Another point worth mentioning is that the docker daemon would not start on my Windows VM when set to use linux containers, which has something to do with nested virtualization (running a VM inside a VM.) Since I don’t really care about linux containers at the moment, I simply switched to windows containers, and slowly backed into the bushes.
Now that our client is configured to trust our insecure registry, let’s try the push again.
PS C:\> docker push 10.211.55.2:5000/colorpicker
The push refers to a repository [10.211.55.2:5000/colorpicker]
242b7a69a21f: Pushed
0281ebf270fd: Pushed
4777122753b8: Pushed
de57d9086f9a: Skipped foreign layer
f358be10862c: Skipped foreign layer
latest: digest: sha256:7d3ff34231abec035813d36f1cfe48c737e15ba9b6db9dd238e5c1f166bcb73d size: 1570
Success!