The move to Kubernetes – when our system administrator Thomas heard about these plans, he was immediately excited. "Just before Christmas 2018, the developers came to me with this spontaneous idea," he says. "We had to work flat out to procure and install servers as well as install Kubernetes on them." That all happened really quickly and then: "We worked as a team, all the Sysadmins tackled it together and got it done in the desired time frame." Shortly after Christmas, the system was up and running on the new servers.
But what is Kubernetes? It is an open-source platform that automates the operation of Linux containers. Groups of hosts are combined into clusters which can then be managed easily and efficiently, largely without manual adjustment. Kubernetes is particularly well suited for all web applications that require rapid scaling.
Fast response
As a music store with Europe's largest warehouse, a stable Kubernetes distribution is important. Because it has to be suitable for production too. We use OpenShift to make the web store function smoothly even with increased search queries. "At the moment, Kubernetes is in use for Thomann's mobile app," Thomas says. On the app, customers search for instruments in categories such as "electric guitar," "electric bass" or "drums." "If the mobile app is sluggish, it doesn't leave a good impression with users," he says. "Navigation needs to work instantly." Even during peak loads such as Christmas, Kubernetes can quickly rescale containers, he adds. "It wouldn't be easy to make this possible with the old system – with virtual and physical servers – and it wouldn’t be anywhere like as quick."
Thomas joined our subsidiary Bits & Beats in early 2018. Before that, he was a Senior Systems Engineer at a Nuremberg-based service provider. He even then working with OpenShift. "When I started at Bits & Beats, I immediately grabbed a few decommissioned servers and said: Let's build an OpenShift cluster with the old servers just for fun," he says. He held a Kubernetes training session for his system administration colleagues. "We built the cluster with OpenShift. That's how it all started."
Thomas explains how he migrates a project to Kubernetes step by step: "First, I look what servers the project is running on. Then I go to the servers and have a look: What operating system is this? What version of PHP is it? And what components are running on it?" Those components then need to be converted to the new servers. Thomas creates a Git Repository to centrally manage the versions of the system software. He then puts everything in there that is needed to deploy the software later. Since the configuration can be very similar depending on the type of application, a Template Git Repository is used to draw from. "The application that was running on the server before can then run in Kubernetes in a similar way," he says. "When the migration is done, the DNS record will be redirected. The URL then points to the Kubernetes project and no longer to the old server." This makes the project live and accessible externally.
The objective: Make it as stable as possible
The Sysadmin Team at Bits & Beats was already able to gain a lot of experience with the test cluster. Shortly before Christmas 2018, when we wanted to use Docker containers as part of a new project ourselves, everyone was well prepared. The project members set up physical servers in Nuremberg first. We also rented resources for a productive cluster in the Google Cloud in Frankfurt. These virtual machines can be started at the push of a button. For the productive cluster, we selected the Frankfurt 3 zone, which is divided into availability zones A, B and C. In each availability zone we placed one Master Server, one Infra Server and one Worker Node. This distribution is intended to ensure the best possible stability. "In theory, an entire availability zone (a datacenter) - could fail," says Thomas, "And yet the cluster would continue to run."
The distribution between physical servers (Nuremberg) and virtual servers (Google, Frankfurt) makes sense. "Everything that is strongly connected to Thomann's databases runs on the Nuremberg servers," says Thomas. "The rest of the project sites run in the Google Cloud." On the Nuremberg servers, Kubernetes manages the load distribution. A so-called Scheduler distributes the workload. In the event of a load peak, e.g. when many users are using the store at the same time, the Horizontal Pod Autoscaler - another Kubernetes component - comes into play and starts additional containers.
But what exactly are containers? A compact description can be found on redhat.com. "A Linux container is a set of one or more processes isolated from the rest of the system." All files necessary for execution are provided via an image. "Linux containers are always portable and consistent from development to testing to execution," redhat.com states. "They are therefore much faster than development pipelines that replicate traditional test environments."
According to Thomas, this straightforwardness was one of the main reasons why we chose Docker containers and thus Kubernetes: "The necessary libraries and services are always attached to the application. As a result, the whole thing also runs smoothly on a notebook. This benefits the developers."
From base image to deployment
Thomas explains how he works with containers. "The containers build on each other. That means you have a base image, for example an Alpine. You can then pack SSL certificates into it, so that all later applications also have them. On this basis, you can then build PHP versions 7.2 and 7.3, for example. And the application uses this in turn." At the end, during the CI/CD run, separate Docker images are created. (There will be more on Docker containerization technology in our next Kubernetes blog post.)
"All the containerization and deployment of the application is done automatically," Thomas says. "The only thing done manually is debugging, aka looking for errors. Meanwhile, the Kubernetes resources we need which tell the cluster what state we want, come from a Helm chart." That resides in the Git Repository. The versioning system Git facilitates software development: It allows developers to monitor and revert their own changes and make them available to other developers via repositories. "Everyone usually has their own Git Fork, which is kind of a clone of the main project," Thomas says. "This allows everyone to work simultaneously and merge their work via pull requests."
For a new project, developers are assigned the subfolder of an existing Git Repository or a new Git Repository. "That's where they start with their code," Thomas explains. "Usually they test the code locally first, on their laptop. Then, when they progress, the Continuous Delivery System turns it into a container and eventually rolls it out." Our company uses a CI/CD system called Drone CI to do this. "The Drone server takes the software, builds it, downloads libraries along with it, packages everything into the actual Docker image and pushes it into a registry," Thomas explains. “The very last step is the deployment, which means the project is rolled out in Kubernetes."
Distributing to multiple network nodes
"In production, we run the containers multiple times," Thomas says. "Kubernetes make the containers run as far apart as possible. For example, if an application is scaled to two containers, those two containers don't run on ONE physical node, but always on different ones." So if one of those nodes fails, it still works. By the way, during maintenance, the sysadmins empty the node: "This is called draining. Once the node is empty, we can use it any way we want."
As mentioned earlier, the Thomann Kubernetes runs in two locations: Nuremberg and Frankfurt. The differences in servers affect running costs in the cloud. "In the Google Cloud, we use virtual machines that you have to pay for by the minute," Thomas says. "When scaling these VMs, we get pretty close to the needed resource limit." In Nuremberg however, Thomann has strong physical servers "so that we have enough wiggle room," as Thomas puts it. Hence the company is prepared for peak loads.
Kubernetes itself and everything around it is also constantly being developed. "A new trend is serverless," says Thomas, explaining the principle: "Applications that I only need for a limited time can be implemented perfectly on a serverless platform. The application is made available. As soon as a request arrives, the system acts dynamically, starts the container, executes the application and then immediately stops the container again." Such a serverless platform can be created in Kubernetes with Knative, for example.
In addition, there are a lot of different operators now that are able to fully automatically deploy an XtraDB cluster in Kubernetes. Such an operator is a program that also runs in Kubernetes. It knows the applications very well, configures them fully automatically and thus extends the capabilities of the cluster. "There are virtually no limits to what we can do," Thomas explains with delight.
Conclusion
Thomas thinks that the greatest strength of Kubernetes is its flexibility. There is the ability to run the system locally on the notebook or to deploy it to another public cloud in an emergency for example. "Other advantages include better scalability and leaner servers," Thomas says. "Version management via Git Repository also makes the whole thing very transparent."
By the way, at Thomann, the migration from the old server structure to Kubernetes is far from complete, says Thomas: "It's all still work in progress." And for the entire Thomann Team, it's an exciting challenge and a big opportunity.
