What is Docker and why need it?

We found ourselves in a situation where we needed a solution to address compatibility issues while also allowing us to make modifications to individual components without impacting others or tinkering with the underlying operating system. Enter Docker, a game-changer.

With Docker, we can run each component in its separate container, complete with its own dependencies and libraries, all within the same virtual machine and operating system. The magic here is that while these components run side by side, they do so in isolated environments, ensuring that changes made to one container won't affect others or require alterations to the host operating system. It's like having your own sandbox for each piece of software, ensuring compatibility and flexibility without compromising the integrity of the overall system.

What are containers?

Containers provide a fully isolated environment, complete with their own processes, networking interfaces, and file systems. Think of them as virtual machines, but here's the twist: they all share the same underlying OS kernel. Containers aren't new; they've been around for about a decade. Some examples of container technologies include LXC, LXD, and LXCFS. Docker harnesses the power of LXC containers. Setting up these container environments can be quite challenging because they operate at a low level. This is where Docker steps in as a high-level tool, offering a range of powerful functionalities that make life easier for end-users like us.
Consider popular operating systems like Ubuntu, Fedora, SUSE, or CentOS. They all consist of two key elements: an OS kernel and a set of software packages. The OS kernel handles interactions with the hardware, while the software above it varies. This software includes user interfaces, device drivers, compilers, file managers, developer tools, and more. So, essentially, they all share a common Linux kernel but differentiate themselves through their unique software stacks.
Let's say you have a system running Ubuntu with Docker installed. Docker can run containers based on different Linux distributions like Debian, Fedora, SUSE, or CentOS. How? It's because all these distributions are based on the same underlying Linux kernel. Docker containers have additional software that makes each OS unique, and Docker makes use of the underlying kernel of the host, which works seamlessly with all these different OS variations. However, there's a catch - Windows doesn't share this same Linux kernel. Therefore, you can't run a Windows-based container on a Docker host running Linux. To achieve that, you'd need Docker on a Windows server. When you run a Linux container on Windows, Windows essentially runs it as a Linux virtual machine under the hood. But is this a disadvantage? Not really, because Docker isn't meant to virtualize different OSes and kernels on the same hardware. Its primary purpose is to package, containerize, and ship applications, allowing them to run consistently across various environments as many times as needed.

Containers Vs Virtual Machines

Containers and virtual machines (VMs) are both virtualization technologies, but they differ in important ways:

Isolation

VMs provide full isolation. Each VM runs a complete operating system and has its own kernel, CPU, memory, etc. This provides a strong security boundary between VMs.
Containers provide lightweight isolation. They share the host operating system kernel and run in user mode. This makes them more resource-efficient but with less isolation between containers.

Operating System

VMs run a full guest operating system, making them more resource intensive. But they can run any operating system.
Containers only run the user mode portion of an OS and the apps. They use fewer resources but must run the same OS as the host.

Deployment

VMs are deployed individually using tools like Hyper-V Manager or at scale using tools like vSphere.
Containers are deployed individually using Docker or at scale using an orchestrator like Kubernetes.

Using Containers and Virtual Machines Together: The Best of Both Worlds

Each has its strengths and unique capabilities, but what happens when you bring them together? It's like combining the best of both worlds to create a dynamic and versatile ecosystem for your applications. Let's explore why using containers and virtual machines together can be a winning strategy.

Enhanced Security

Robust Isolation: Virtual machines (VMs) offer strong isolation between themselves and the host system. This isolation is achieved by running each VM on a hypervisor, which acts as a protective boundary. If one VM is compromised due to a security vulnerability or a malicious attack, it is contained within its isolated environment, unable to affect other VMs or the host system.
Isolated Containers: By running containers inside VMs, you create an additional layer of security. Even if a container within a VM is compromised, it remains confined within that VM. This containment prevents lateral movement, minimizing the potential damage caused by security breaches.
Multi-Tenant Environments: In scenarios where multiple users or applications share the same infrastructure, such as in cloud computing, using VMs to isolate containers ensures that one user's or application's actions don't impact others. Each VM serves as a security boundary.

Unparalleled Flexibility

Custom Configurations: Virtual machines can be individually customized to suit specific requirements. Each VM can have its unique configuration, including different operating systems, software stacks, and resource allocations. This flexibility is valuable when you need to accommodate diverse application needs.
Legacy and Modern Compatibility: Running containers inside different VMs allows you to bridge the gap between legacy applications and modern ones. Legacy applications that require older operating systems or dependencies can coexist with modern applications, all within their own tailored VM environments.
Testing and Development: Developers can set up VMs with varying configurations to replicate different production environments for testing. This helps ensure that applications work seamlessly across a range of setups before deployment.

Optimized Resource Allocation

Resource Guarantees: Virtual machines enable precise resource allocation. You can allocate specific amounts of CPU, memory, and other resources to each VM. This means that critical applications can be guaranteed the resources they need to operate efficiently, regardless of the resource demands of other VMs.
Resource Scaling: VMs also allow for dynamic scaling of resources. You can adjust resource allocations as needed, scaling up or down to meet changing demands. This ensures optimal resource utilization without wasting capacity.

Versatile Operating System Options

Operating System Compatibility: Containers within VMs give you the flexibility to run different operating systems and versions on a single host system. This is particularly useful for testing and compatibility scenarios. For instance, you can run containers with Linux and Windows operating systems on the same host, facilitating compatibility testing across platforms.
Version Control: VMs can host different versions of the same operating system, allowing you to test applications across various OS versions. This helps identify any compatibility issues and ensures that applications are robust across multiple environments.

Docker Image

An image in the world of Docker is like a blueprint, much like a virtual machine (VM) template you might be familiar with in the virtualization world. It serves as the foundation to create one or more containers, which are running instances of these images. Containers are like isolated environments; they run independently, each with its own set of processes and environments.

What's fascinating is that we can craft our own Docker images and make them accessible to others through a Docker registry, a bit like sharing a recipe.

In the traditional software development process, developers would create applications and then hand them over to operations teams to deploy and manage in a production environment. This handoff often involved a set of complex instructions, specifying how the hosting environment should be configured, what prerequisites were required, and how dependencies should be set up.

This process could be challenging for the operations team because they didn't develop the application, and setting it up exactly as the developers intended could be tricky. Any issues that arose often required collaboration between developers and operations to resolve.

With Docker, the lines between development and operations blur. DevOps teams collaborate closely to transform these deployment instructions into something called a 'Dockerfile.' This Dockerfile encapsulates both the developer's requirements and the operations team's specifications.

The result is an image that contains everything needed to run the application. This image can be deployed on any host system with Docker installed, ensuring the application runs consistently and smoothly, just as it's supposed to.

This is a perfect example of how Docker contributes to the DevOps culture. It brings developers and operations together, streamlining the deployment process, and making it easier to maintain and scale applications in a unified way.

Commands for working with containers

The docker run command is used to create and start a new container from an image. It has many options to configure the container.

Some useful options are:

p or -publish to expose a port
v or -volume to mount a volume
e to set an environment variable
-name to assign a name
-rm to automatically remove the container when it exits

Example:

docker run -p 80:80 --name my-nginx -d nginx

This will run an nginx container exposing port 80, with the name my-nginx in detached mode.

The docker ps command lists running containers. You can use options like:

a to show all containers, not just running
q to only show IDs
-format to format the output
f or -filter to filter by conditions

Example:

docker ps -a -f status=exited

This will show all exited containers.

The docker stop command stops a running container. You can stop a container by name or ID.

Example:

docker stop my-nginx

This will stop the my-nginx container.

The docker rm command removes a container. You can pass multiple container names or IDs.

Example:

docker rm my-nginx another-container

This will remove 2 containers.

So in summary, the basic workflow is:

docker run to create and start a container
docker ps to check running containers
docker stop to stop a running container
docker rm to remove the container

docker rmi IMAGERemoves one or more images from your system. You need to remove all tags referencing an image before you can delete it.

docker rmi myimage:latest

the docker pull command pulls an image from a registry like Docker Hub. If the image does not exist locally, it will be downloaded.

docker pull nginx

You can attach to a running container using:

docker attach CONTAINER

Or execute a command inside a running container using:

docker exec CONTAINER COMMAND [ARG...]

For example:

docker exec mycontainer ls -l

This will list files inside the mycontainer container.

You can run containers in:

Attached mode - Connected to your terminal
Detached mode - Runs in the background

To detach from an attached container, you can use:

CTRL + P + Q - Default detach keys
Custom detach keys set using -detach-keys

RUN - STDIN

Imagine you have a simple command-line application that, when run, asks for your name and then prints a welcome message based on your input. If you decide to containerize this application and run it as a Docker container, you might encounter an unexpected behavior. Instead of waiting for your input, the container immediately prints the welcome message, without letting you provide your name.

This behavior occurs because, by default, Docker containers do not listen to standard input (stdin). Even though you are connected to the container's console, it doesn't read your input. The container runs in a non-interactive mode, assuming it doesn't have any input to process.

To make it possible to provide input to the container, you need to explicitly map the standard input of your host to the Docker container using the '-i' parameter. The '-i' flag stands for interactive mode. However, even with the '-i' flag, you might still face issues with prompts not appearing correctly.

The reason for this is that the application expects a terminal to display the prompt, but we haven't attached to the container's terminal. To make this work seamlessly, you should also use the '-t' flag, which stands for a pseudo-terminal. By combining '-i' and '-t', you not only attach to the container's terminal but also enable interactive mode within the container.

This combination of '-i' and '-t' ensures that you can interact with the application inside the container effectively, including providing input as expected."

RUN - PORT Mapping

Suppose you're running a web application within a Docker container on your Docker host. The Docker host, or Docker engine, is the underlying system where Docker is installed. When the containerized web application runs, you can see that the server is up and running. Now, the question is: how can a user access your application?

Your application is listening on a specific port, let's say port 5000. To access it from your web browser, you have a couple of options:

Using the Container's IP: Every Docker container is assigned an IP address by default. However, this IP is internal and can only be accessed from within the Docker host. It's not reachable for users outside the host.
Using the Docker Host's IP: To make your application accessible to users outside the Docker host, you can use the IP address of the Docker host itself. For example, the Docker host's IP might be 192.168.1.5. But for this to work, you need to map the port from inside the Docker container to a free port on the Docker host. You can achieve this using the '-p' parameter in your 'docker run' command.

For instance, if you want users to access your application through port 80 on the Docker host, you can map port 80 on the host to port 5000 on the Docker container. This way, multiple instances of your application can be mapped to different ports on the Docker host, allowing for flexibility and scalability.

RUN - Volume Mapping

Imagine you're running a MySQL container. When you create databases and tables inside this container, the data files are stored in a specific location, typically /var/lib/mysql within the Docker container. It's important to note that Docker operates in an isolated environment, so any changes made to files happen exclusively within the container.

Now, let's say you've inserted a lot of data into the database, but you decide to delete the MySQL container. When you remove the container, all the data inside it is deleted as well. To preserve your data, you can map a directory from outside the container on the Docker host to a directory inside the container.

For instance, you can use the following command to run a MySQL container and map an external directory /opt/datadir to the /var/lib/mysql directory inside the container:

docker run -v /opt/datadir:/var/lib/mysql mysql

With this setup, when the Docker container runs, it automatically mounts the external directory to a folder inside the container. This ensures that all your data is stored in an external volume and remains intact even if you delete the Docker container.

If you want to retrieve detailed information about a Docker container, including its state, mounts, configuration, network settings, and more, you can use the docker inspect command followed by the container's name or ID, like this:

docker inspect blissful_hopper

To access the logs generated by a container, which are essentially the contents written to its standard output, you can use the docker logs command. Simply specify the container's name or ID, like so:

docker logs blissful_hopper

Why and how to create your own docker image?

Imagine you're building a house of Lego blocks, but you need a special, unique piece that you can't find in the ready-made sets. That's when you decide to create your very own Lego piece. In the world of software and containers (like digital boxes for software), Sometimes, the specific component or service you need for your application may not be readily available on Docker Hub, or if you and your team have decided to dockerize your application for easier shipping and deployment. In such cases, you have to create your own Docker image. So, you make your own!

Creating your custom Docker image involves:

Planning: Start by considering the steps you would follow if you were to deploy the application manually. Document these steps in the correct order. This planning is crucial.
Step-by-Step Instructions: For instance, if you were setting up the application manually, you might begin with an operating system like Ubuntu. Then, you'd update the source repositories using the 'apt' command, install necessary dependencies with 'apt,' install Python dependencies using 'pip,' copy your application's source code to a specific location (e.g., '/opt'), and finally, launch the web server using the 'flask' command.
Create a Dockerfile: With these instructions in hand, you can create a Dockerfile (Explained in next section). This file outlines how to set up your application inside a Docker container. You'll write down each step from your manual deployment process in the Dockerfile.
Building the Image: Once you've created the Dockerfile, you use the 'docker build' command to build your Docker image. Specify the Dockerfile as input and provide a tag name for the image. This step creates the image locally on your system.
Publish to Docker Hub (Optional): If you want to make your custom image available to others, you can push it to the Docker Hub registry. Use the 'docker push' command, specifying the name of the image you just created. This allows others to use your image for their Docker deployments.

Dockerfile

A Dockerfile is like a set of cooking instructions written in a special format that Docker can easily understand. It's a bit like giving your computer a recipe to cook up a perfect dish.

In this Dockerfile, you list out all the ingredients (software packages), describe how to prepare them (commands to run), and even specify how to serve the dish (how your application should run). Docker uses this file to create a special package, your Docker image, which contains everything your application needs to run smoothly.

Here's what you typically include in a Dockerfile:

Base Image: Think of this as the basic type of dish you're making. You start with a known starting point, like a type of cuisine. This is often an existing image from Docker Hub.
Instructions: These are like cooking steps. You tell Docker what to do, like adding specific ingredients (installing software), using certain tools (commands), and arranging everything just right (setting up configurations).
Final Dish: At the end of your Dockerfile, you specify what the finished dish should look like when it's served (how your application should run).

Once your Dockerfile is ready, you use it to build your Docker image. Then, you can use this image to create containers (instances) that run your application consistently, no matter where they are. It's like having a reliable recipe that ensures your dish turns out perfect every time you cook it.

Layered Architecture in Docker

When Docker builds images, it does it in a clever way, using something called a "layered architecture." Imagine building a sandwich with different layers – each layer adds something unique to the sandwich.

Here's how it works in Docker:

Layer by Layer: Every line of instruction in a Dockerfile creates a new layer in the Docker image. Each layer contains changes from the previous one. It's a bit like adding ingredients to your sandwich step by step.
Efficient Storage: What's cool is that each layer only stores what's different from the previous one. This helps keep things efficient and saves space. Think of it as only adding new ingredients to your sandwich without repeating what's already there.
Layer Sizes: If you look at a Docker image, you'll notice that some layers are larger, like the base image (think of it as the bread), while others are smaller (like the toppings). This reflects in the image's size.
See the Layers: You can check out these layers with the docker history command. It's like peeking at the layers of your sandwich to see what's inside.

Docker Build Output

When you run the docker build command, you'll see a list of steps involved in building your image, almost like watching a chef prepare your sandwich. What's nifty is that Docker caches these layers. So, if something goes wrong (like a dropped pickle), you can restart the build from that step without redoing everything.

Handling Failures and Updates

Imagine if, during the sandwich-making process, you drop a tomato. In Docker terms, if a step (like installing software) fails, you fix it and rerun the build. Docker cleverly reuses the layers that didn't change, saving you time. The same goes for adding new steps to your Dockerfile; you don't need to rebuild everything.

This layered approach is handy when you're updating your software or making frequent changes. You only need to redo the parts that actually changed, just like you'd only replace the tomato if it fell, not the whole sandwich. It keeps things efficient and quick.

Let's use an example to illustrate Docker's layered architecture:

Imagine you're building a digital sandwich, which is actually a web application, using Docker. Here's how it works:

Step 1: The Base Layer - Bread You start with a plain piece of bread, representing the base image. This bread is like the foundation of your sandwich. It's a Docker image, let's call it "BaseOS," which is about 120MB in size. It's just the essential stuff to run a system.

Step 2: Adding Ingredients - Software Packages Now, you want to add some software packages to your digital sandwich, like adding lettuce, tomatoes, and cheese to your physical sandwich. Each package is like an ingredient. You run commands in your Dockerfile to install these packages, and they add layers to your image.

Lettuce (300MB): This layer contains the lettuce (software package) you added. It adds about 300MB to the image size.
Tomatoes (Small Layer): The tomatoes (another package) don't add much to the image size because Docker is smart and only stores what's different from the previous layer. So, it's a small layer.
Cheese (Small Layer): Similarly, adding cheese (yet another package) results in a small layer.

Step 3: The Final Dish - Your Application Now, you're ready for the main part, your web application (the sandwich filling). You copy your application's code into the image and configure how it should run. This layer represents your app.

Viewing the Layers You can use the docker history command to see all these layers, just like checking the layers of a sandwich. You'll notice the base image, the software packages, and your application.

Handling Updates Let's say you want to add pickles to your sandwich, which means adding more software to your app. You only need to create a new layer for the pickles, not redo the whole sandwich. Docker is like a chef who can efficiently update your digital sandwich without wasting ingredients or time.

So, Docker's layered approach allows you to create efficient, reusable, and easily updatable "sandwiches" (or software) for your projects. It's like being able to change the toppings on your sandwich without making a new one from scratch!

Docker Compose

Imagine you're not just making a sandwich but an entire meal with multiple dishes. You want everything to come together perfectly. That's where Docker Compose steps in.

What is Docker Compose?

Docker Compose is like having a master recipe book for your meal. It's a tool that helps you set up and manage complex applications that consist of multiple services, like a web server, a database, and more. Instead of juggling multiple docker run commands, you create a configuration file in YAML format called docker-compose.yaml.

Setting Up the Master Recipe

In your docker-compose.yaml, you list all the services your application needs and specify options for each one. It's like listing all the dishes you want for your meal and the ingredients for each dish.

Running the Meal

Now, the magic happens when you run the docker-compose up command. It's like having a skilled chef who follows your master recipe and prepares the entire meal for you. Docker Compose brings up all the services together, ensuring they work seamlessly as a team.

Easier, Efficient, and Tidy

Using Docker Compose makes setting up complex applications easier, more efficient, and easier to maintain. It's like having a chef's assistant who handles all the cooking, so you don't have to worry about each dish separately.

Single Docker Host Limitation

However, it's important to note that Docker Compose is typically used for running containers on a single Docker host. It's like preparing a meal in one kitchen. If you need to spread your dishes across multiple kitchens (or hosts), you might need other tools like Kubernetes to orchestrate everything.

In summary, Docker Compose is your go-to tool when you want to orchestrate multiple containers and services into a cohesive application, simplifying the setup and management of complex software projects on a single Docker host. It's like having a chef's assistant who follows your recipes to prepare a fantastic meal effortlessly!

Let's use a real-world example

Imagine you want to create a blogging platform with a web server, a database, and a content management system (CMS). Docker Compose will help you set up and manage this multi-service application.

Create a Docker Compose File (docker-compose.yaml):

version: '3'

services:
  webserver:
    image: nginx:latest
    ports:
      - "80:80"
  database:
    image: mysql:latest
    environment:
      MYSQL_ROOT_PASSWORD: mysecretpassword
      MYSQL_DATABASE: blogdb
  cms:
    image: wordpress:latest
    ports:
      - "8080:80"
    environment:
      WORDPRESS_DB_HOST: database
      WORDPRESS_DB_USER: root
      WORDPRESS_DB_PASSWORD: mysecretpassword
      WORDPRESS_DB_NAME: blogdb

In this example, we have three services: webserver (Nginx for web hosting), database (MySQL for data storage), and cms (WordPress for the content management system). The YAML file specifies images, ports, and environment variables for each service.

Running the Application:

To bring up the entire application stack, use the following command in the directory where your docker-compose.yaml file is located:

docker-compose up

Docker Compose will start the specified services and link them together, just like a chef assembling various dishes for a complete meal.

Access Your Blog:

Once the services are up and running, you can access your blog in a web browser:

Web Server: http://localhost
WordPress CMS: http://localhost:8080

Managing the Application:

To stop the application (containers), press Ctrl + C in the terminal where you ran docker-compose up. This won't remove the containers; it just stops them.
To stop and remove the containers, use the following command:

docker-compose down

This real-world example demonstrates how Docker Compose simplifies the setup and management of multi-service applications. You define your services and their configurations in a docker-compose.yaml file, run a single command to start everything, and easily manage your application stack. It's like having a recipe for a multi-course meal, where Docker Compose ensures all the dishes are prepared and served harmoniously.

Docker Engine

When we talk about Docker Engine, we're essentially talking about a host system with Docker software installed. When you install Docker on a Linux host, you're actually installing three essential components:

The Docker Daemon: This is like the behind-the-scenes manager. It runs quietly in the background, handling all the important Docker objects like images, containers, volumes, and networks. Think of it as the conductor orchestrating a symphony.
The Docker REST API Server: Imagine this as the way you communicate with the Docker Daemon. It's like a special phone line that lets other programs talk to the Daemon and give it instructions. You can even create your own tools that use this interface to control Docker.
The Docker Command Line Interface (CLI): This is your direct line of communication with Docker. It's a command-line tool that allows you to do things like start containers, stop containers, delete images, and much more. The CLI uses the REST API to talk to the Docker Daemon. What's cool is that the Docker CLI doesn't have to be on the same computer as the Daemon. You can have it on another system, like your laptop, and still control a remote Docker Engine. Just use the -h option in the Docker command and specify the address and port of the remote Docker Engine.

Namespace?

In Linux, a namespace is a feature that provides a way to create isolated and separate instances of certain system resources. Each namespace encapsulates a specific resource, such as processes, network interfaces, filesystems, and more, allowing processes within a namespace to operate as if they were in their own isolated environment. Namespaces are a fundamental building block for containerization and resource isolation in modern operating systems.

The primary goal of namespaces is to provide process-level isolation, where processes within one namespace cannot directly access or affect resources in other namespaces. This isolation is essential for creating lightweight and isolated environments, especially in scenarios like containerization.

Here are a few common types of namespaces in Linux:

PID Namespace (Process ID Namespace): As explained earlier, PID namespaces isolate process IDs, allowing processes in different namespaces to have their own separate process ID spaces. This prevents processes in one namespace from interfering with processes in other namespaces.
Network Namespace: Network namespaces isolate network-related resources such as network interfaces, IP addresses, routing tables, and firewall rules. Processes in different network namespaces can have their own isolated networking setups, effectively creating virtual network stacks.
Mount Namespace: Mount namespaces provide isolated filesystem mount points. This allows processes to have their own independent view of the filesystem hierarchy, so changes to mounts in one namespace do not affect others.
UTS Namespace: UTS namespaces isolate the system's hostname and domain name. This allows processes in different UTS namespaces to have different hostname and domain settings, which can be useful for creating separate system identities within containers.
IPC Namespace (Inter-Process Communication Namespace): IPC namespaces isolate certain inter-process communication mechanisms like System V IPC (e.g., semaphores, shared memory, message queues). This ensures that processes in different namespaces cannot directly communicate through these mechanisms.
User Namespace: User namespaces isolate user and group identifiers. This is particularly useful for running processes with different user and group IDs within containers, providing better security and isolation.
Cgroup Namespace (Control Group Namespace): Cgroup namespaces allow processes to have different views of the control groups hierarchy. This is important for resource management and limiting the amount of resources processes can use.

These namespaces, among others, collectively provide the means to create isolated environments that operate as if they were separate instances of the entire operating system. Containerization technologies like Docker, Kubernetes, and LXC heavily utilize namespaces to create lightweight and isolated environments for running applications.

How does docker make use of namespace?

Docker uses namespaces to create separate environments for various aspects such as process IDs, networking, interprocess communication, file systems, and system time. This isolation ensures that containers operate independently and do not interfere with each other.

Let's delve into one of the ways Docker isolates processes using namespaces – the Process ID (PID) namespace:

1. Global Namespace (Initial Boot): When Linux boots up, it begins in what we call the global namespace. Here, all processes share a common pool of process IDs. The very first process, known as the init process, receives the ID 1. It's like the elder ancestor of all other processes on the system.

2. Creating a New PID Namespace: As the system starts, it's possible to create new PID namespaces. This is particularly useful in containerization technologies like Docker, where each container should have its own isolated world. When a new PID namespace is born, a process inside it becomes the new init process, starting with PID 1 in that specific namespace.

3. Process IDs in a New PID Namespace: Processes within a fresh PID namespace possess their own unique set of process IDs. These IDs are distinct from those in the global namespace and other PID namespaces. For instance, if you have two containers, Container A and Container B, each in its own PID namespace, both containers can have a process with PID 1. However, these PID 1s are distinct within their respective namespaces, preventing any clashes with one another or with the global namespace.

4. Process Hierarchy in a New Namespace: Much like in the global namespace, a PID namespace maintains a hierarchy. Child processes within a namespace have PIDs specific to that namespace and appear as offspring of their parent process within that same namespace. This structure maintains order and organization within the isolated environment.

5. Isolation and Termination: Processes within a PID namespace cannot see processes in other namespaces, whether they are in different PID namespaces or in the global namespace. When a PID namespace is dismantled, all processes within that namespace are terminated. This occurs because, from the global namespace's perspective, those processes are left orphaned.

CGROUPS

We've learned that the Docker host and its containers share the same system resources like CPU and memory. But how much of these resources are allocated to containers, and how does Docker handle this sharing?

By default, there are no resource limits, meaning a container could potentially consume all the resources available on the underlying host. However, Docker provides a way to control how much CPU and memory each container can use. It accomplishes this through a technology called control groups (cgroups), which restrict the hardware resources allocated to containers.

CPU Resource Management:

You can limit the CPU usage of a container by using the --cpus option with the docker run command. For example, providing a value of 0.5 will ensure that the container doesn't consume more than 50% of the host's CPU at any given time.

Memory Resource Management:

Similar to CPU, you can control a container's memory usage. Docker allows you to set limits on how much memory a container can use. This prevents a container from hogging all available memory and affecting the overall system's performance.

In summary, Docker uses control groups (cgroups) to manage and allocate system resources like CPU and memory to containers. This ensures that containers operate within defined resource limits, preventing them from monopolizing the host's resources and ensuring a balanced and efficient environment.

Docker Storage

Docker containers are fantastic for running applications, but they often need to store data as well. Docker provides several storage options to manage data effectively within containers. Here's how Docker storage works:

Container Filesystem: Each Docker container has its own isolated filesystem. It's like having a dedicated space where the container can read and write files. This filesystem starts from a base image and stores any changes or additions made during container runtime. However, this filesystem is ephemeral, meaning it disappears when the container is removed. It's perfect for temporary data, but not for long-term storage.

Container Volumes: Sometimes, you need to persist data beyond the lifecycle of a single container. That's where volumes come in. Volumes are like shared folders between containers and the host system. They allow data to be stored outside the container filesystem, ensuring it remains intact even if the container is removed. Volumes are often used for databases, configuration files, or shared data between containers.

Bind Mounts: Bind mounts are a way to share data between the host and containers. They are like windows into the host filesystem. With bind mounts, you can access and manipulate files on your host system from within a container, and vice versa. Bind mounts are useful for scenarios where you want to share files or directories between your local machine and a container, making development and debugging easier.

Storage Drivers: Docker uses storage drivers to manage the way it interacts with the host system's storage. Different operating systems and filesystems may require different storage drivers. Popular storage drivers include Overlay2 (used by default on many Linux distributions), Device Mapper, and Btrfs. These drivers handle tasks like image layering and container data management efficiently.

Docker Compose and Persistent Storage: When orchestrating multi-container applications with Docker Compose, you can define volumes in your docker-compose.yaml file to ensure that data remains persistent across container restarts. This is crucial for applications like content management systems or databases, where data integrity is paramount.

Storage Plugins: Docker offers a plugin system that allows you to extend its storage capabilities. These plugins integrate with external storage solutions such as cloud storage providers, network-attached storage (NAS) devices, or distributed storage systems. These plugins enable advanced data management and backup strategies for Docker containers.

Let's dive deeper into a real scenario where Docker storage is used

Imagine you're running a web application with a MySQL database in Docker containers. You want to ensure that your MySQL data remains intact even if you update or recreate your MySQL container. This is crucial because losing your database data would result in data loss and downtime for your application.

Here's how you can set up this scenario:

Step 1: Create a Docker Volume

First, create a Docker volume using the docker volume create command. You can give it a meaningful name, like mydbdata:
```
  docker volume create mydbdata
```
This volume acts as a dedicated storage space for your MySQL data.

Step 2: Run the MySQL Container

Next, run the MySQL container using the docker run command. In this command, you'll specify the --name to name your container, --env to set environment variables like the MySQL root password, and -v to mount the volume you created into the container:
```
  docker run --name mysql-container -e MYSQL_ROOT_PASSWORD=mysecretpassword -v mydbdata:/var/lib/mysql -d mysql:latest
```
- --name mysql-container: Assigns the name "mysql-container" to your MySQL container for easy reference.
- -e MYSQL_ROOT_PASSWORD=mysecretpassword: Sets the root password for MySQL within the container.
- -v mydbdata:/var/lib/mysql: Mounts the mydbdata volume into the /var/lib/mysql directory inside the container. This is where MySQL stores its data files.

Step 3: Use the MySQL Container

You can now use the MySQL container as you would in any application. Your database data is stored within the mydbdata volume. This means even if you stop, remove, or recreate the MySQL container, your data remains intact in the volume.

Step 4: Data Persistence

As you continue to use your application, any data changes made within the MySQL container are saved in the volume. The data is persistently stored, and you can perform regular backups of the volume to ensure data safety.

This scenario demonstrates how Docker volumes allow you to separate your data from your containers, ensuring data persistence and preventing data loss even in dynamic containerized environments. It's a robust solution for applications that rely on databases or require long-term data storage.

Docker Networking

Docker networking is like setting up a bunch of phones for containers, allowing them to talk to each other and the outside world. It's crucial for organizing containers and keeping them secure.

Here's how it works:

1. Default Network: When Docker is installed, it creates a basic network called "bridge." Containers on this network can talk to each other like text messaging between friends. But they can't reach anyone outside unless you tell them how.

2. Custom Networks: Sometimes, you want to group specific containers together, like a private chat group. Docker lets you create custom networks for this purpose. Containers in the same network can talk just like friends in the same chat room.

3. Host Connection: Containers can also join your computer's network, similar to a friend using your phone. They can call anyone your phone can call, but it's less private.

4. Across Multiple Computers: In more complex setups with many computers, Docker has a feature called "overlay networks." It's like setting up a virtual meeting room where containers from different computers can talk to each other as if they were in the same room.

5. Special Phone Numbers: Docker can even give each container its unique phone number (IP address) on the network. It's like assigning a distinct phone number to each person in a group call.

6. Connecting to the Internet: If a container wants to access the internet, it can either use your computer's internet or share it. Imagine a container borrowing your phone's internet to surf the web.

7. Naming Friends: Docker helps containers remember each other's names, so they don't have to remember phone numbers. It's like having a phonebook for containers.

8. Staying Safe: Docker ensures that containers only talk to authorized friends, like controlling who can call whom.

In simpler terms, Docker networking is all about making sure containers can communicate within their group and with the world outside. It offers various ways to set up these conversations, just like managing a phone system for a big group of friends.

Inspecting a network

Inspecting a Docker network allows you to gather detailed information about its configuration and status. To inspect a Docker network, follow these steps:

Open your terminal or command prompt.
Use the following command to inspect a Docker network, replacing your_network_name with the actual name of the network you want to inspect:
```
 docker network inspect your_network_name
```
For example, if you have a network named "my_app_network," you would run:
```
 docker network inspect my_app_network
```

After running the command, Docker will provide a JSON-formatted output containing information about the specified network. This information includes details like the network's name, ID, driver, and the containers attached to it.

Here's an example of what the output might look like:

 [
     {
         "Name": "my_app_network",
         "Id": "abc12345def67890",
         "Created": "2023-10-05T12:34:56.789012345Z",
         "Scope": "local",
         "Driver": "bridge",
         "EnableIPv6": false,
         "IPAM": {
             "Driver": "default",
             "Options": {},
             "Config": [
                 {
                     "Subnet": "172.18.0.0/16",
                     "Gateway": "172.18.0.1"
                 }
             ]
         },
         "Internal": false,
         "Containers": {
             "container1_id": {
                 "Name": "container1",
                 "EndpointID": "xyz98765...",
                 "MacAddress": "02:42:ac:12:00:02",
                 "IPv4Address": "172.18.0.2/16",
                 "IPv6Address": ""
             },
             "container2_id": {
                 "Name": "container2",
                 "EndpointID": "pqr54321...",
                 "MacAddress": "02:42:ac:12:00:03",
                 "IPv4Address": "172.18.0.3/16",
                 "IPv6Address": ""
             }
         },
         "Options": {
             "com.docker.network.bridge.default_bridge": "true",
             "com.docker.network.bridge.enable_icc": "true",
             "com.docker.network.bridge.enable_ip_masquerade": "true",
             "com.docker.network.bridge.host_binding_ipv4": "0.0.0.0",
             "com.docker.network.bridge.name": "docker0",
             "com.docker.network.driver.mtu": "1500"
         },
         "Labels": {}
     }
 ]

This output provides a wealth of information about the network, including its driver, subnet configuration, and containers connected to it.

That's how you inspect a Docker network to get insights into its settings and connections.

Embedded DNS

Embedded DNS, often referred to as "DNS resolution," is a feature in Docker that plays a crucial role in enabling containerized applications to communicate with each other using domain names instead of IP addresses. It simplifies the process of network communication within Docker containers and is an integral part of Docker networking.

Here's how Embedded DNS works and its role in Docker networking:

Name Resolution: Docker containers can communicate with each other using domain names instead of relying solely on IP addresses. This means you can refer to other containers using user-friendly names that are easier to remember and manage. For example, if you have two containers, "web" and "database," you can configure them to communicate by name, such as "web" connecting to "database," instead of using IP addresses.
Automatic DNS Configuration: Docker automatically sets up an embedded DNS server for each custom bridge network and attaches it to that network. When containers are attached to the same network, they can resolve each other's domain names using this DNS server. This DNS server knows about all the containers connected to the network and their IP addresses, enabling seamless name resolution.
Built-in DNS Names: Docker assigns a DNS name to each container based on its container name. For example, if you have a container named "webapp" attached to a network, you can refer to it using the DNS name "webapp" within that network.
Cross-Network Resolution: Docker's DNS resolution allows containers on the same network to resolve each other's names. Additionally, if you have multiple custom bridge networks, containers can resolve the names of containers in other networks using their DNS names.
Integration with Service Discovery: Embedded DNS is a fundamental component of service discovery within Docker networks. It simplifies the process of discovering and connecting to services running in different containers. Tools like Docker Compose and Docker Swarm rely on DNS resolution to ensure containers can communicate across services.

Docker Registry

A Docker registry is a central repository for storing and distributing Docker images. It serves as a hub where Docker users can share and download container images, making it a fundamental component of the Docker ecosystem. Docker images are lightweight, stand-alone, and contain everything needed to run an application, making them an efficient way to package and distribute software.

Why to use it?

Image Distribution: A registry allows you to distribute your Docker images to others easily. Whether you're sharing your application with a team, the open-source community, or deploying it to production, a registry ensures that the required images are readily accessible.

Version Control: Registries typically support versioning of images. This means you can have multiple versions of the same image, allowing you to roll back to a previous version if needed. This version control is crucial for maintaining consistency in different environments.

Security: Docker registries often provide features for securing and controlling access to your images. You can set up authentication, authorization, and access control policies to ensure that only authorized users can pull or push images.

Offline Access: Once images are downloaded from a registry, they can be used in environments with limited or no internet connectivity. This is valuable for deploying containers in isolated or air-gapped environments.

Setting it up

Setting up a Docker registry involves choosing between using a public registry or deploying a private registry.

Public Registry: Docker Hub is the most well-known public Docker registry. It's a cloud-based service where you can publish your public images for free. To use Docker Hub, you need to create an account, log in, and follow the instructions to push your images. Public registries are great for sharing open-source projects and images that don't require strict access control.

Private Registry: If you need more control and security, you can set up your private Docker registry. Docker provides an official registry image called Docker Registry (also known as Docker Distribution). To set up a private registry, you can follow these steps:

Pull the Docker Registry Image: Use the following command to pull the Docker Registry image from Docker Hub:
```
 docker pull registry:latest
```
Run the Registry Container: Create a Docker container from the image you pulled:
```
 docker run -d -p 5000:5000 --name my-registry registry:latest
```
This command starts the private registry container and exposes it on port 5000. You can choose a different port if needed.
Push and Pull Images: Now, you can push your Docker images to your private registry using the docker push command and pull them from there using docker pull.
```
 docker push my-registry:5000/my-image:tag
 docker pull my-registry:5000/my-image:tag
```

Remember to secure your private registry with authentication and access control to prevent unauthorized access.

Orchestration

Container orchestration refers to the automated management and coordination of containers, particularly in the context of deploying and scaling containerized applications. It involves tasks such as provisioning containers, distributing workloads across multiple container instances, ensuring high availability, managing container lifecycle (starting, stopping, scaling), load balancing, and handling failures. The goal of container orchestration is to simplify the deployment and management of containerized applications, making them more resilient, scalable, and efficient.

Why it is recommended?

Scalability: Orchestration platforms can automatically scale the number of container instances up or down based on demand. This ensures that your application can handle varying levels of traffic without manual intervention.

High Availability: Orchestration frameworks can distribute containers across multiple nodes (physical or virtual machines) to ensure that your application remains available even if some nodes or containers fail.

Load Balancing: Orchestration tools can manage load balancing, ensuring that incoming requests are distributed evenly among containers to optimize performance and prevent overloading.

Resource Optimization: Orchestration platforms can allocate resources efficiently, making the most of available CPU, memory, and storage resources, which helps reduce infrastructure costs.

Rolling Updates and Rollbacks: Container orchestration allows for seamless updates and rollbacks of application versions. This minimizes downtime and reduces the risk of deploying faulty code.

Service Discovery: Orchestration frameworks typically include service discovery mechanisms that allow containers to find and communicate with each other, even as containers are added or removed.

Configuration Management: Container orchestrators enable centralized configuration management, making it easier to update application settings and environment variables across all containers.

Security: Orchestration tools often include security features such as network segmentation, access controls, and image scanning to enhance the security of containerized applications.

Portability: Orchestration platforms abstract the underlying infrastructure, making it easier to move applications between different cloud providers or on-premises data centers.

Popular container orchestration platforms include Kubernetes, Docker Swarm, and Apache Mesos. Kubernetes, in particular, has gained widespread adoption due to its robust feature set and large community support.

How it works?

When you use a container orchestration tool like Kubernetes, you'll define an application's configuration using either a YAML or JSON file. This configuration file instructs the management tool on where to locate container images, how to set up networking, and where to store logs.

When you initiate a new container deployment, the management tool automatically arranges the deployment within a cluster and selects the appropriate host, considering any specified requirements or limitations. The orchestration tool then oversees the container's lifecycle according to the directives outlined in the compose file.

Kubernetes offers patterns that enable you to manage the configuration, lifecycle, and scaling of container-based applications and services. These reusable patterns serve as the building blocks for Kubernetes developers to construct complete systems.

Container orchestration is applicable in various environments, including on-premises servers, public clouds, and private cloud setups, wherever containers are utilized.

Docker Simplified

A Comprehensive Guide to the Basics

Table of contents

What is Docker and why need it?

What are containers?

Containers Vs Virtual Machines

Isolation

Operating System

Deployment

Using Containers and Virtual Machines Together: The Best of Both Worlds

Enhanced Security

Unparalleled Flexibility

Optimized Resource Allocation

Versatile Operating System Options

Docker Image

Commands for working with containers

RUN - STDIN

RUN - PORT Mapping

RUN - Volume Mapping

Why and how to create your own docker image?

Dockerfile

Layered Architecture in Docker

Let's use an example to illustrate Docker's layered architecture:

Docker Compose

Let's use a real-world example

Docker Engine

Namespace?

How does docker make use of namespace?

CGROUPS

Docker Storage

Let's dive deeper into a real scenario where Docker storage is used

Docker Networking

Inspecting a network

Embedded DNS

Docker Registry

Why to use it?

Setting it up

Orchestration

Why it is recommended?

How it works?

Resources