Vectops

Mikrotik: Save your config via FTP

Sat, 08 Mar 2025 00:00:00 +0000

Do you have a MikroTik? Have you ever made changes to the configuration and then, after a while, when you reset the configuration, it didn’t work as you expected?

Don’t worry, if you have an FTP server at home, we can solve this quickly and easily.

I’m going to tell you a very simple way (really, a very simple way) to make FTP backups of your MikroTik and keep your configurations safe.

Download the script from our repository: ftp_backup

Configure the script

Create the script in System -> Scripts menu, click on Add new, and fill in the form fields as follows:

- Name: ftp_backup
- Policy: read, write, test

In the script, variables are defined through the :local refs where you need to put the information:

:local FTPserver "x.x.x.x"
:local FTPuser "username"
:local FTPpassword "password"
:local FTPdestFile "remote_directory"

Just fill the spaces between double commas for each variable with the values you got in the previous step.

Now, you can test the script by applying and clicking on Run Script, or via cli:

/system scripts run ftp_backup

Scheduling the script

Now here’s the thing. We got everything set up, verified that runs as intended, and everything is ready. But at this point, if you don’t click on the “Run script” button, the script won’t trigger itself. Hopefully that is quite simple to fix: just configure a Scheduler to run this script on a given time basis (say 1d).

To achive this, from your Mikrotik’s web UI, go to System -> Scheduler menu, and click on Add new, then just fill each field with the next details:

- Name: ftp_backup
- Interval: 1d
- Start time: 23:00:00
- Policy: read, write, test
- On event: /system script run ftp_backup

If you prefer setting this up through the cli, this is done just by running the following command:

/system scheduler
add interval=1d name=ftp_backup on-event="/system script run ftp_backup" policy=read,write,test start-time=23:00:00

Remember that you can change the interval accordingly to fit your own needs.

With that, we can say we’re done! You have configured the FTP backup on your Mikrotik, and now you can break anything in your config, you have a backup!

See you next time, don’t forget to share & leave a comment!

Flysky (iBUS) interfacing to an arduino

Thu, 27 Feb 2025 00:00:00 +0000

Recently I had a need to control an arduino with an RC Controller, this was a proof of concept project that took 2 or maybe 3 tries to get working but from what I’ve seen online there’s not that much documentation for projects like this.

This is a very sparse and simple explanation for the process but hey, it should work.

This project, as well as other electronics projects I do use PlatformIO on VSCode because the Arduino IDE and I don’t get along. Even the newer 2.0 one.

Required parts

Flysky FS-A8S:

Breadboard:

Flysky FS-i6 RC Controller:

Arduino PRO Micro:

Wiring

There’s only 3 wires to take into account here:

Arduino Pro Micro PIN	FS-A8S PIN
RX1(D0)	IBus
Ground	Ground
5v(VCC)	Vin

Setting up the Arduino Pro Micro

The Pro Micro’s setup is pretty straight forward, since we’re using a FlySky RC Controller for this setup why not use it’s iBUS protocol wich is widely documented and easy to implement on the arduino since it already has a library. You can find the library here along with it’s documentation.

Platformio need to have a platformio.ini file, this is created automagically when you create the project using platformio’s interface -> next, next, next, confirm, wait, profit.

Once it’s created you can set up libraries and other stuff on it, since this is a basic proof of concept we’re only importing the iBusBM library:

[env:sparkfun_promicro16]
platform = atmelavr
board = sparkfun_promicro16
framework = arduino
lib_deps = bmellink/IBusBM @ ~1.1.4

As for the main.cpp arduino code (it’s heavily commented for your pleasure):

// Include iBusBM Library
#include <IBusBM.h>
#include <Arduino.h>

// Create iBus Object
IBusBM ibus;

// Read the number of a given channel and convert to the range provided.
// If the channel is off, return the default value
int readChannel(byte channelInput, int minLimit, int maxLimit, int defaultValue) {
  uint16_t ch = ibus.readChannel(channelInput);
  if (ch < 100) return defaultValue;
  return map(ch, 1000, 2000, minLimit, maxLimit);
}

// Read the channel and return a boolean value
bool readSwitch(byte channelInput, bool defaultValue) {
  int intDefaultValue = (defaultValue) ? 100 : 0;
  int ch = readChannel(channelInput, 0, 100, intDefaultValue);
  return (ch > 50);
}

void setup() {
  // Start serial monitor
  Serial.begin(115200);

  // Attach iBus object to serial port
  ibus.begin(Serial1);
}

void loop() {

  // Cycle through first 5 channels and determine values
  // Print values to serial monitor
  // Note IBusBM library labels channels starting with "0"

  for (byte i = 0; i < 5; i++) {
    int value = readChannel(i, -100, 100, 0);
    Serial.print("Ch");
    Serial.print(i + 1);
    Serial.print(": ");
    Serial.print(value);
    Serial.print(" | ");
  }

  // Print channel 6 (switch) boolean value
  Serial.print("Ch6: ");
  Serial.print(readSwitch(5, false));
  Serial.println();

  delay(10);
}

Then just build it and upload it onto the Pro Micro

Setting up the RC Controller

Just bind it, same procedure as 10 years ago.

Additional Considerations

This “sketch” outputs the values incomming from 6 channels, if you have another RC Controller that has more channels just change the loop that goes through every channel.

Once you have those values you can parse them into variables of your own definition. Take this as a starting point. Baby steps!

2025 blog update: Late new year's resolution

Wed, 26 Feb 2025 00:00:00 +0000

Hello,

We’ve been busy with real work, family, family work, working families, robotics? family robots! and so on but we haven’t forgotten about this blog/site.

We’ve decided to take a quick turn so the site gets atleast some updates. (Beleive it or not, there’s people that actually follow this besides the 3 of us….. I know right?)

Since this site was originally meant for very specific problems that arose whenever we had an issue at work or testing something out on our homelab setups, it hinders what we can post without “contaminating” everything else that has been posted but… as you can probably tell, we don’t maintain this site for profit, just for shits and/or giggles. No responsibilities to any higher power other than the people who read this, the half a dozen of you who do.

This means that we can basically do whatever the hell we want with it, not that anyone’s complaining (try it, I double dare you).

So, from my part, I have several ongoing projects that involve electronics, robotics, and several other weird things that interest a lot of people online (atleast 2). I’ll try to provide updates on them, start to…. maybe finish.

From my counterparts on Vectops, I’m pretty sure that if I start posting regularly they will be inspired to do so too(hopefully).

I guess i’ll see you around. I just have to get started, how hard can it be? baby steps!

2024 blog update: farewell WordPress, hello Hugo

Sun, 24 Dec 2023 00:00:00 +0000

Hello there!

Just a heads up from Vectops. It’s been a while since our last post here, but we’re far from done with the blog! What a busy year has 2023 been for everybody, huh?

Anyway, we’re pleased to announce that our scope for the project towards the upcoming year is to continue bringing interesting and helpful approaches to common issues that anyone in the IT world can face.

For now, the big news is that we have recently switched from the good ol’ WordPress engine to Hugo so the whole site is now managed as code, something we as authors are already thankful for. Hopefully this change will bring more stability, better performance and even help reducing the attack surface of our platform, something essential for us as webmasters. All without sacrificing useful features for you as readers, such as the “search” bar or the archive.

The only thing we still have to re-think is the comment system, since it is the main way you as user got to interact with us and our content. It’s a pity that all the old comments had to be left inside WordPress, since there was some valuable insights from you as well over there, but don’t worry, we’ll eventually come up with a solution for this.

Wish you all the best, and let’s keep rolling in this next year.

The Vectops team

DevSecOps: Integrating Security in DevOps - A Comprehensive Guide

Mon, 08 May 2023 00:00:00 +0000

In the modern software development landscape, security is more important than ever. DevSecOps, the practice of integrating security into the DevOps process, aims to ensure that applications are developed and deployed with security in mind from the very beginning.

In this comprehensive guide, we’ll explore the fundamentals of DevSecOps, explain how to integrate security into your DevOps workflow, and provide examples to help you get started.

Outline

Understanding DevSecOps
  a. What is DevSecOps?
  b. DevSecOps principles
  c. Benefits of DevSecOps

Integrating Security in the DevOps Pipeline
  a. Shift-Left Security
  b. Continuous Security Testing
  c. Security as Code
  d. Automated Security Monitoring

DevSecOps Tools and Practices
  a. Static Application Security Testing (SAST)
  b. Dynamic Application Security Testing (DAST)
  c. Container Security
  d. Infrastructure as Code (IaC) Security
  e. Secrets Management

Building a DevSecOps Culture
  a. Collaboration and Communication
  b. Security Awareness and Training
  c. Continuous Improvement

DevSecOps Examples
  a. Integrating SAST into the CI/CD pipeline
  b. Implementing container security scanning
  c. Secure IaC practices

Understanding DevSecOps

What is DevSecOps?

DevSecOps is the practice of integrating security into the DevOps process. It aims to address security concerns throughout the entire software development lifecycle, from planning to deployment. DevSecOps promotes a proactive approach to security, ensuring that security is built into applications from the very beginning, rather than being added as an afterthought.

DevSecOps Principles

The core principles of DevSecOps include:

Collaboration: Encourage communication and collaboration between development, security, and operations teams.
Automation: Automate security processes and tools within the DevOps pipeline to improve efficiency and reduce human error.
Shift-Left Security: Integrate security early in the development process to identify and address vulnerabilities sooner.
Continuous Improvement: Continuously evaluate and improve security practices and tools based on feedback and lessons learned.

Benefits of DevSecOps

Implementing DevSecOps offers several advantages, such as:

Faster remediation: By addressing security issues early in the development process, vulnerabilities can be fixed more quickly and efficiently.
Reduced risk: Integrating security into the DevOps pipeline reduces the risk of security breaches and data loss.
Improved collaboration: DevSecOps fosters better communication and collaboration between development, security, and operations teams, leading to more secure and reliable applications.
Cost savings: By detecting and fixing security issues early, organizations can avoid the high costs associated with security breaches and remediation.

Integrating Security in the DevOps Pipeline

To integrate security into your DevOps pipeline, consider adopting the following practices:

Shift-Left Security

“Shift-Left” security means incorporating security practices early in the development process. This can include activities such as threat modeling, secure coding practices, and code reviews focused on security.

Example: During the design phase, conduct threat modeling sessions with your team to identify potential risks and vulnerabilities in your application. Address these risks by implementing secure coding practices and validating the security of your code through regular code reviews.

Continuous Security Testing

Continuously test your application for security vulnerabilities throughout the development process. This can involve static application security testing (SAST), dynamic application security testing (DAST), and interactive application security testing (IAST).

Example: Integrate a SAST tool, such as SonarQube or Checkmarx, into your CI/CD pipeline to automatically analyze your code for security vulnerabilities during the build process.

Security as Code

Define and manage security policies, configurations, and controls as code. This approach allows you to version control security settings, automate security policy enforcement, and improve the auditability of security configurations.

Example: Use a tool like HashiCorp Sentinel or Open Policy Agent to enforce security policies programmatically across your infrastructure.

Automated Security Monitoring

Automate the monitoring and analysis of your application and infrastructure for security events and anomalies. This can include tools for log analysis, intrusion detection, and security information and event management (SIEM).

Example: Configure a SIEM tool, such as Splunk or Elasticsearch, to collect and analyze security-related logs from your application and infrastructure. Set up alerts to notify your team of potential security incidents. DevSecOps Tools and Practices

Static Application Security Testing (SAST)

SAST tools analyze your source code for potential security vulnerabilities without executing the code. Popular SAST tools include SonarQube, Checkmarx, and Fortify.

Example: Integrate SonarQube into your CI/CD pipeline to automatically analyze your code for security vulnerabilities during the build process.

Dynamic Application Security Testing (DAST)

DAST tools analyze your running application for security vulnerabilities by simulating attacks and identifying potential issues. Popular DAST tools include OWASP ZAP, Burp Suite, and Acunetix.

Example: Integrate OWASP ZAP into your CI/CD pipeline to automatically perform dynamic security testing on your application during the staging or pre-production phase.

Interactive Application Security Testing (IAST)

IAST tools combine elements of SAST and DAST by analyzing application behavior during runtime to identify security vulnerabilities. Popular IAST tools include Contrast Security, HCL AppScan, and Micro Focus Fortify.

Example: Incorporate Contrast Security into your application to continuously monitor its behavior during development and testing, identifying and alerting on potential security issues in real-time.

Container Security

As containerization becomes more prevalent, it’s essential to ensure the security of your containers and container orchestration platforms. Container security tools help you scan container images for vulnerabilities, enforce security policies, and monitor runtime behavior. Popular container security tools include Aqua Security, Sysdig Secure, and Anchore.

Example: Integrate Aqua Security into your CI/CD pipeline to scan container images for vulnerabilities, enforce security policies, and monitor container runtime behavior.

Infrastructure as Code (IaC) Security

With the increasing adoption of IaC, it’s crucial to ensure the security of your infrastructure code. IaC security tools help you analyze your infrastructure code for misconfigurations, compliance violations, and potential security risks. Popular IaC security tools include Checkov, Terrascan, and Bridgecrew.

Example: Integrate Checkov into your CI/CD pipeline to scan your Terraform or CloudFormation code for security vulnerabilities and compliance issues before deploying infrastructure changes.

By adopting DevSecOps practices and integrating security into your DevOps pipeline, you can create a more secure, efficient, and collaborative development environment. This proactive approach to security will help you reduce risk, improve application reliability, and ultimately protect your organization from potential security breaches.

Building a DevSecOps Culture

To successfully implement DevSecOps, it’s essential to foster a culture of shared responsibility for security across your organization. Here are some tips for building a DevSecOps culture:

Encourage collaboration: Break down silos between development, operations, and security teams by encouraging open communication, collaboration, and shared goals. Regularly hold cross-functional meetings, workshops, and training sessions to enhance understanding and cooperation.
Provide training and resources: Equip your team members with the necessary knowledge and skills to handle security concerns effectively. Offer training, resources, and tools that help them understand security best practices, threat modeling, and secure coding techniques.
**Shift security left:**Integrate security early in the development process by incorporating security requirements into the planning and design phases. This proactive approach will help you identify and address security issues before they become more challenging and costly to fix later in the development lifecycle.
Automate security testing: Use automated security testing tools to streamline the security validation process and ensure that security checks are consistently applied throughout the development pipeline.
Measure and improve: Continuously monitor and measure your DevSecOps efforts by tracking key performance indicators (KPIs), such as vulnerability detection rates, mean time to remediation, and deployment frequency. Use these metrics to identify areas for improvement and refine your DevSecOps practices over time.

By promoting a DevSecOps culture within your organization, you’ll create an environment where security becomes an integral part of the development process, and everyone shares the responsibility for protecting your applications and infrastructure.

Conclusion

DevSecOps represents a significant shift in how organizations approach application and infrastructure security, emphasizing the need to integrate security throughout the entire development lifecycle. By adopting DevSecOps principles and practices, you can enhance collaboration, streamline security validation, and ultimately build more secure, resilient applications that protect your organization from potential security threats.

As you continue to explore DevSecOps, remember to foster a culture of shared responsibility for security, invest in training and resources, and leverage automation to improve the efficiency and effectiveness of your security efforts. With a solid foundation in DevSecOps, your organization will be well-positioned to tackle the ever-evolving security challenges of the modern software development landscape.

Mastering Git: Branching, Merging & Conflicts - A Comprehensive Guide

Wed, 03 May 2023 00:00:00 +0000

Welcome to our in-depth guide on mastering Git, focusing on branching, merging, and handling conflicts. Git is a powerful version control system widely used by developers and teams to manage their codebases efficiently.

Understanding how to work with branches, merge changes, and resolve conflicts is essential for seamless collaboration and effective software development.

Outline

1. Introduction to Git Branching
2. Creating and Switching Branches
3. Merging Branches
4. Handling Merge Conflicts
5. Advanced Branching Techniques
6. Git Best Practices

1. Introduction to Git Branching

Branching in Git allows you to create parallel lines of development, isolating features, bug fixes, or experiments from the main codebase. The default branch in a Git repository is called the “master” branch or, more recently, the “main” branch.

Benefits of Git Branching:

Isolation of work: Developers can work independently on different features or fixes without affecting the main codebase.
Easier collaboration: Team members can share branches to collaborate on specific tasks or review code before merging it into the main branch.
Flexible release management: Teams can maintain separate branches for different release versions, making it easier to manage releases and apply hotfixes.

2. Creating and Switching Branches

Creating a new branch:

To create a new branch and switch to it, use the following command:

git checkout -b feature-xyz

This command creates a new branch called feature-xyz and switches to it.

Switching between branches:

To switch to an existing branch:

git checkout branch-name

Replace branch-name with the name of the branch you want to switch to.

3. Merging Branches

To merge changes from one branch into another, follow these steps:

a. Switch to the target branch (usually the main branch):

git checkout main

b. Merge the source branch into the target branch:

git merge feature-xyz

This command merges the feature-xyz branch into the main branch.

4. Handling Merge Conflicts

Merge conflicts occur when changes in different branches conflict with each other. Git will automatically merge changes whenever possible, but sometimes manual intervention is required.

Resolving merge conflicts:

Identify conflicting files by looking for the CONFLICT message in the Git output.
Open the conflicting files and look for conflict markers («««<, =======, and »»»>), which indicate the conflicting changes.
Review the changes and decide which version to keep, or modify the file as needed to resolve the conflict.
Stage the resolved files using git add file-name.
Commit the changes using git commit.

5. Advanced Branching Techniques

Rebasing:

Rebasing is an alternative to merging that keeps a linear commit history. To rebase a branch onto another branch, use the following command:

git rebase target-branch

Cherry-picking:

Cherry-picking allows you to selectively apply commits from one branch to another. To cherry-pick a commit, use the following command:

git cherry-pick commit-hash

Replace commit-hash with the hash of the commit you want to cherry-pick.

6. Git Best Practices

Use descriptive branch names related to the task or feature.
Keep branches short-lived and focused on a specific task.
Regularly pull changes fromthe main branch to keep your feature branches up-to-date.
Perform small, frequent commits with clear and informative commit messages.
Use pull requests and code reviews to ensure high-quality code before merging into the main branch.

By mastering Git branching, merging, and conflict resolution, you’ll be well-prepared to handle complex projects and collaborate effectively with your team. Implementing Git best practices will further improve your software development process and enhance the overall quality of your codebase. As you continue to hone your Git skills, you’ll find it to be an indispensable tool in managing your projects and streamlining your workflow.

Docker Essentials: Images, Containers & More

Sun, 30 Apr 2023 00:00:00 +0000

In this tutorial, we’ll cover the essentials of Docker, focusing on key concepts like images, containers, and Dockerfiles. By the end of this guide, you’ll have a solid understanding of Docker fundamentals and be prepared to create, manage, and deploy containerized applications with ease.

Article Outline

Introduction to Docker
Key Docker Concepts
  a. Docker Images
  b. Docker Containers
  c. Dockerfiles
  d. Docker Registries
Installing Docker
Creating Your First Docker Image and Container
  a. Writing a Dockerfile
  b. Building the Docker Image
  c. Running a Docker Container
Basic Docker Commands
  a. Listing Images and Containers
  b. Starting, Stopping, and Restarting Containers
  c. Removing Images and Containers
Conclusion

Part 1: Introduction to Docker

Docker is an open-source platform that automates the deployment, scaling, and management of containerized applications. Docker allows you to package an application and its dependencies into a lightweight, portable container that can run consistently across different environments. This consistency simplifies application development, testing, and deployment, making Docker a popular choice among developers and sysadmins.

Part 2: Key Docker Concepts

To get started with Docker, it’s important to understand its key concepts:

a. Docker Images: An image is a lightweight, portable, and executable snapshot of a containerized application and its dependencies. Images are built from a Dockerfile and can be stored in a registry for sharing and deployment.

b. Docker Containers: Containers are instances of Docker images that run the packaged application in an isolated environment. Containers provide consistency and portability across various platforms and environments.

c. Dockerfiles: A Dockerfile is a script that contains instructions for building a Docker image. It defines the base image, application code, dependencies, and configuration required to create a containerized application.

d. Docker Registries: A Docker registry is a centralized repository for storing and sharing Docker images. Docker Hub is the default public registry, but you can also use private registries or create your own.

Part 3: Installing Docker

To start using Docker, download and install the appropriate version for your operating system from the official Docker website. Docker Desktop is available for Windows and macOS, while Docker Engine is available for Linux distributions.

Part 4: Creating Your First Docker Image and Container

Once Docker is installed, you can create your first Docker image and container by following these steps:

a. Writing a Dockerfile: Create a new text file named “Dockerfile” (without any file extension) in your project directory. Write the necessary instructions to define your containerized application, such as specifying the base image, copying application files, installing dependencies, and setting up the runtime environment.

For example: (Note that the html code has been embedded within the Dockerfile for simplicity’s sake)

# Use the official NGINX image as the base image
FROM nginx:latest

# Set the working directory
WORKDIR /usr/share/nginx/html

# Create a simple "Hello, World!" HTML file
RUN echo '<!DOCTYPE html><html><head><title>Hello, World!</title></head><body><h1>Hello, World!</h1></body></html>' > index.html

# Expose the default NGINX port (80)
EXPOSE 80

# Start the NGINX server
CMD ["nginx", "-g", "daemon off;"]

b. Building the Docker Image: In your project directory, run the docker build command followed by the -t flag (to tag the image with a name) and a period (to indicate the current directory). This command builds a Docker image from your Dockerfile.

For the above example, this would be:

docker build -t hello-world-nginx .

c.** Running a Docker Container:** Use the docker run command followed by the image name to create and start a new container from your Docker image. Optionally, you can specify flags to customize the container’s behavior, such as -p to map container ports to host ports, or –name to assign a custom name to the container.

From the previous step, you would use:

docker run -d -p 8080:80 --name hello-world-nginx-container hello-world-nginx

Finally open your browser to the URL: http://localhost:8080 and see the Hello World.

Part 5: Basic Docker Commands

Here are some basic Docker commands to help you manage images and containers:

a. Listing Images and Containers: Use docker images to list all available images on your system, and docker ps (or docker container ls) to list all running containers. To view all containers, including stopped ones, add the -a flag to the docker ps command.

b. Starting, Stopping, and Restarting Containers: To start, stop, or restart a container, use the docker start, docker stop, or docker restart commands, followed by the container ID or name.

c. Removing Images and Containers: Use docker rmi followed by the image ID or name to remove a Docker image. To remove a container, use docker rm followed by the container ID or name. Remember that you’ll need to stop a container before you can remove it.

Part 6: Conclusion

Congratulations! You’ve now learned the essentials of Docker, including its key concepts, how to create images and containers, and basic Docker commands. With a solid foundation in Docker, you’ll be well-equipped to manage and deploy containerized applications effectively, and take your DevOps and sysadmin career to new heights.

As you become more comfortable with Docker, consider exploring additional features like Docker Compose, Docker Swarm, and Docker networking to further enhance your container management skills. Keep learning and experimenting with new Docker features and best practices to stay ahead in the rapidly evolving world of containerization.

Kubernetes Basics: Cluster Setup & Deployment

Sun, 23 Apr 2023 00:00:00 +0000

In this tutorial, we’ll walk you through the basics of Kubernetes, focusing on setting up a Kubernetes cluster and deploying a containerized application. By the end of this guide, you’ll have a solid understanding of Kubernetes fundamentals and be ready to manage containerized applications with ease.

Article Outline

Introduction to Kubernetes
Kubernetes Components
  a. Nodes
  b. Pods
  c. Services
  d. Deployments
Setting Up a Kubernetes Cluster
  a. Using a managed Kubernetes service
  b. Setting up a cluster manually
Deploying a Containerized Application
  a. Creating a deployment
  b. Exposing the application using a service
  c. Scaling the application
  d. Updating the application
Conclusion

Part 1: Introduction to Kubernetes

Kubernetes is an open-source container orchestration platform that automates the deployment, scaling, and management of containerized applications. Developed by Google and now maintained by the Cloud Native Computing Foundation (CNCF), Kubernetes has become the industry standard for container orchestration, supporting a wide range of container runtimes, including Docker and containerd.

Part 2: Kubernetes Components

Kubernetes uses various components to manage containerized applications:

a. Nodes: These are the worker machines that run containers. Nodes can be virtual or physical machines, and a Kubernetes cluster can consist of one or more nodes.

b. Pods: The smallest and simplest unit in Kubernetes, pods are used to deploy containers. A pod can contain one or more containers that share the same network namespace and storage volumes.

c. Services: A Kubernetes service is an abstraction that defines a logical set of pods and a policy to access them. Services are used to expose applications running in pods to the network, either within the cluster or externally.

d. Deployments: Deployments are used to declaratively manage the desired state of your containerized application. They can automatically manage and update pods based on the specified configuration.

Part 3: Setting Up a Kubernetes Cluster

To set up a Kubernetes cluster, you have two options:

a. Using a managed Kubernetes service: Many cloud providers offer managed Kubernetes services, such as Google Kubernetes Engine (GKE), Amazon Elastic Kubernetes Service (EKS), and Azure Kubernetes Service (AKS). These services simplify cluster setup and management by automating tasks like node provisioning and updates.

b. Setting up a cluster manually: For more control over your cluster, you can set it up manually using tools like kubeadm, Minikube, or Kops. This approach requires more hands-on management but offers greater customization.

Part 4: Deploying a Containerized Application

Once your Kubernetes cluster is set up, follow these steps to deploy a containerized application:

a. Creating a deployment: Write a YAML manifest file to define your application’s deployment, specifying the container image, desired replicas, and other configuration details. Use the kubectl apply command to create the deployment.

For example:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: myapp-deployment
spec:
  replicas: 3
  selector:
    matchLabels:
      app: myapp
  template:
    metadata:
      labels:
        app: myapp
    spec:
      containers:
      - name: myapp-container
        image: myapp-image:v1
        ports:
        - containerPort: 8080

b. Exposing the application using a service: To make your application accessible, create a Kubernetes service by defining a YAML manifest file and applying it with kubectl apply. The service will expose the application on a specific port or load balancer, depending on the chosen service type.

apiVersion: v1
kind: Service
metadata:
  name: myapp-service
spec:
  selector:
    app: myapp
  ports:
    - protocol: TCP
      port: 80
      targetPort: 8080
  type: LoadBalancer

c. Scaling the application: To scale your application, update the replicas field in your deployment manifest, and reapply the configuration using kubectl apply. Kubernetes will automatically create or remove pods to match the desired replica count.

# Update the replicas field in your deployment manifest
# For example, change the replicas from 3 to 5, then run:

kubectl apply -f myapp-deployment.yaml

d. Updating the application: To update your application, modify the container image or other configuration details in your deployment manifest, and reapply the updated manifest using kubectl apply. Kubernetes will perform a rolling update, replacing old pods with new ones while maintaining the desired replica count and minimizing downtime.

# Update the container image or other configuration details in your deployment manifest
# For example, change the image from myapp-image:v1 to myapp-image:v2, then run:

kubectl apply -f myapp-deployment.yaml

Part 5: Conclusion

Congratulations! You’ve now learned the basics of Kubernetes, including its key components, setting up a cluster, and deploying a containerized application. As you become more comfortable with Kubernetes, you can explore additional features like persistent storage, ingress controllers, and custom resource definitions to further enhance your container orchestration skills.

For more basic Kubernetes information I highly recommend you read: Phippy goes to the zoo by the Cloud Native Computing Foundation. Yes, it looks like a kid’s book but it clears out a lot of decent information.

By mastering Kubernetes basics, you’ll be well-equipped to manage and scale containerized applications efficiently, and take your DevOps and sysadmin career to new heights. Don’t forget to keep learning and experimenting with new Kubernetes features and best practices to stay ahead in the rapidly evolving world of container orchestration. update, replacing old pods with new ones

Beginner's Guide to Infrastructure as Code (IaC) - Part 5: Advanced IaC Techniques and Future Trends

Tue, 18 Apr 2023 00:00:00 +0000

In the final part of our beginner’s guide to Infrastructure as Code, we’ll explore advanced IaC techniques and examine future trends in the IaC landscape. By mastering these advanced concepts, you’ll be well-equipped to stay ahead in the rapidly evolving world of Infrastructure as Code.

Article Outline

Advanced IaC techniques
  a. Policy as Code
  b. Immutable infrastructure
  c. Infrastructure monitoring and observability
  d. IaC refactoring
Future trends in Infrastructure as Code
  a. AI and Machine Learning in IaC
  b. Edge computing and IaC
  c. Serverless infrastructure management

Part 5: Advanced IaC Techniques and Future Trends

Advanced IaC techniques

To further enhance your IaC skills, consider exploring these advanced techniques:

a. Policy as Code: Extend IaC principles to security, compliance, and governance by defining policies as code. Tools like HashiCorp Sentinel, Open Policy Agent, and Azure Policy allow you to enforce policies programmatically across your infrastructure.

Here’s an example, using OPA(Open Policy Agent), with a Rego policy file example_policy.rego

package main

deny[msg] {
    input.resource.kind == "Pod"
    image := input.resource.spec.containers[_].image
    not startswith(image, "example.com/")
    msg := sprintf("Invalid image registry: %v", [image])
}

b. Immutable infrastructure: Adopt an immutable infrastructure approach, where infrastructure components are replaced rather than updated. By using IaC tools to create and manage immutable infrastructure, you can improve consistency, reduce configuration drift, and simplify rollback processes.

Using Terraform, you could add the following to your main.tf:

resource "aws_instance" "web" {
  ami           = "ami-0c55b159cbfafe1f0"
  instance_type = "t2.micro"

  tags = {
    Name = "web-${timestamp()}"
  }
}

c. Infrastructure monitoring and observability: Incorporate monitoring and observability tools into your IaC workflows to gain deeper insights into the performance and health of your infrastructure. Tools like Prometheus, Grafana, and ELK Stack can help you collect, analyze, and visualize infrastructure metrics.

Example Ansible playbook for deploying Prometheus and Grafana:

- name: Deploy Prometheus and Grafana
  hosts: monitoring
  become: yes

  tasks:
    - name: Install Docker
      ansible.builtin.package:
        name: docker
        state: present

    - name: Install docker-compose
      ansible.builtin.package:
        name: docker-compose
        state: present

    - name: Deploy Prometheus and Grafana with docker-compose
      ansible.builtin.docker_compose:
        project_src: /path/to/prometheus-grafana-docker-compose
        state: present

This Ansible playbook deploys Prometheus and Grafana using Docker and docker-compose on a target host labeled monitoring. 
Make sure to replace /path/to/prometheus-grafana-docker-compose with the path to the directory containing the docker-compose.yml file for your Prometheus and Grafana setup.

d. IaC refactoring: Periodically review and refactor your IaC code to improve maintainability, readability, and efficiency. Techniques like modularization, code reuse, and removing duplication can help you keep your IaC codebase clean and up-to-date.

Future trends in Infrastructure as Code

Stay ahead of the curve by keeping an eye on these emerging trends in the IaC landscape:

a. AI and Machine Learning in IaC: As AI and Machine Learning continue to advance, expect to see more intelligent IaC tools that can optimize infrastructure management, detect anomalies, and even suggest code improvements. Read: ChatGPT

b. Edge computing and IaC: With the rise of edge computing, IaC tools will need to adapt to manage the distributed nature of edge infrastructure, enabling efficient deployment and management of edge devices and services.

c. Serverless infrastructure management: The increasing adoption of serverless computing will lead to the development of IaC tools and techniques tailored to manage serverless infrastructure, such as AWS Lambda functions and Azure Functions.

By mastering advanced IaC techniques and staying informed about future trends, you’ll be well-prepared to tackle the challenges of modern infrastructure management. With a solid foundation in Infrastructure as Code, you’ll continue to excel in your DevOps and sysadmin career, and lead your organization towards more efficient and automated infrastructure management.

Beginner's Guide to Infrastructure as Code (IaC) - Part 4: Integrating IaC with CI/CD Pipelines

Mon, 17 Apr 2023 00:00:00 +0000

In Part 4 of our beginner’s guide to Infrastructure as Code, we’ll focus on the integration of IaC into Continuous Integration and Continuous Delivery (CI/CD) pipelines.

You can check the previous part here or if you want to start from the beginning, go here

Integrating IaC into your CI/CD pipeline is crucial for efficient infrastructure management and automation. By using more transition words, we’ll make this part even more readable and engaging.

Article Outline

Understanding CI/CD pipelines
Benefits of integrating IaC with CI/CD pipelines
Steps to integrate IaC into your CI/CD pipeline
  a. Choose a CI/CD platform
  b. Create the pipeline configuration
  c. Add IaC stages to the pipeline
  d. Automate testing and validation
  e. Monitor and optimize the pipeline

Part 4: Integrating IaC with CI/CD Pipelines

Understanding CI/CD pipelines

Continuous Integration (CI) and Continuous Delivery (CD) are practices that enable the automation of building, testing, and deploying code and infrastructure changes. CI involves merging code changes into a shared repository, where automated builds and tests are run. CD, on the other hand, automates the deployment of these changes to various environments, such as staging or production.

Benefits of integrating IaC with CI/CD pipelines

Integrating IaC with your CI/CD pipeline offers several advantages:

Improved collaboration: IaC and CI/CD together promote collaboration among team members by encouraging code reviews and shared ownership of the codebase and infrastructure.
Faster deployments: Automated infrastructure provisioning and deployment through CI/CD pipelines enable faster, more efficient deployments.
Reduced errors: Automation reduces the likelihood of human errors, resulting in more reliable and stable infrastructure.
Easier rollback: If issues arise, you can quickly revert to previous infrastructure versions using version-controlled IaC code and CI/CD rollback mechanisms.

Steps to integrate IaC into your CI/CD pipeline

To integrate IaC into your CI/CD pipeline, follow these steps:

a. Choose a CI/CD platform: Select a CI/CD platform that fits your organization’s needs and is compatible with your chosen IaC tool. Examples of CI/CD platforms include Jenkins, GitLab CI/CD, and AWS CodePipeline.

b. Create the pipeline configuration: Define the pipeline structure and stages using your CI/CD platform’s configuration language, such as Jenkinsfile for Jenkins or .gitlab-ci.yml for GitLab.

For example, using Jenkins with Terraform:

pipeline {
    agent any

    stages {
        stage('Initialize Terraform') {
            steps {
                sh 'terraform init'
            }
        }

        stage('Validate Terraform') {
            steps {
                sh 'terraform validate'
            }
        }

        stage('Plan Terraform') {
            steps {
                sh 'terraform plan'
            }
        }

        stage('Apply Terraform') {
            steps {
                sh 'terraform apply -auto-approve'
            }
        }
    }
}

Another example, using Gitlab CI/CD with Ansible:

stages:
  - prepare
  - validate
  - deploy

prepare:
  stage: prepare
  script:
    - ansible-galaxy install -r requirements.yml

validate:
  stage: validate
  script:
    - ansible-playbook --syntax-check site.yml

deploy:
  stage: deploy
  script:
    - ansible-playbook -i inventory.ini site.yml

c. Add IaC stages to the pipeline: Incorporate IaC-specific stages into your pipeline, such as infrastructure provisioning, configuration management, and deployment. Ensure these stages align with your infrastructure management workflows.

An example using AWS CodePipeline with AWS CloudFormation:

- In your AWS CodePipeline, click "Edit" to modify your pipeline.
- Add a new stage by clicking the "+" icon.
- Name the stage, for example, "Deploy CloudFormation".
- Click "Add action group" in the new stage.
   * Choose "AWS CloudFormation" as the action provider.
   * Select an action mode, such as "Create or update stack".
   * Provide the necessary inputs, such as Stack name, Template file, and any required Parameters.
- Click "Done" to add the action to the stage.

This setup creates a new stage in your AWS CodePipeline that deploys or updates a CloudFormation stack as part of the pipeline.

d. Automate testing and validation: Implement automated testing and validation for your IaC code within the pipeline. This process can include static analysis, unit testing, and integration testing to ensure that your code is functional and adheres to best practices.

An example on how to automate testing with Terraform:

pipeline {
    agent any

    stages {
        // ...
        
        stage('Validate Terraform') {
            steps {
                sh 'terraform validate'
            }
        }

        stage('Terraform Security Check') {
            steps {
                sh 'tfsec .'
            }
        }

        stage('Terraform Compliance') {
            steps {
                sh 'terraform-compliance -f . -p plan.out'
            }
        }
        // ...
    }
}

e. Monitor and optimize the pipeline: Continuously monitor your CI/CD pipeline’s performance and optimize it based on your organization’s needs. Consider factors like deployment frequency, success rate, and the duration of each pipeline stage.

By integrating Infrastructure as Code into your CI/CD pipeline, you’ll streamline your infrastructure management process and improve the overall efficiency of your DevOps and sysadmin workflows. In the next and final part of our beginner’s guide, we’ll explore advanced IaC techniques and future trends to help you stay ahead in the rapidly evolving world of IaC.

Beginner's Guide to Infrastructure as Code (IaC) - Part 3: Implementing IaC Best Practices and Real-World Examples

Sun, 16 Apr 2023 00:00:00 +0000

In this third installment of our beginner’s guide to Infrastructure as Code, we’ll explore the best practices for implementing IaC in your organization.

You can check the previous part here or if you want to start from the beginning, go here

Additionally, we’ll provide real-world examples that demonstrate the power and versatility of IaC. With a focus on readability and SEO, let’s dive into the world of IaC best practices and examples.

Article Outline

IaC best practices
  a. Maintain consistency
  b. Use version control
  c. Modularize your code
  d. Validate and test your infrastructure code
  e. Keep secrets secure
  f. Document your code
Real-world IaC examples
  a. Deploying a web application
  b. Managing a multi-cloud environment
  c. Scaling infrastructure on demand

Part 3: Implementing IaC Best Practices and Real-World Examples

IaC best practices

To ensure your Infrastructure as Code implementation is effective and efficient, consider following these best practices:

a. Maintain consistency: Consistently apply naming conventions, directory structures, and coding styles across your IaC codebase. This consistency makes it easier for team members to collaborate and understand the code.

b. Use version control: Employ a version control system, such as Git, to track changes in your IaC code. This practice allows you to roll back to previous versions if needed and simplifies collaboration among team members.

c. Modularize your code: Break your IaC code into reusable modules, which can be shared across projects. Modularization helps reduce code duplication and makes maintenance more manageable.

d. Validate and test your infrastructure code: Regularly test your IaC code using techniques such as static analysis, unit testing, and integration testing. Testing ensures that your code is functional and adheres to best practices.

e. Keep secrets secure: Store sensitive information, such as API keys and passwords, securely using tools like HashiCorp Vault or AWS Secrets Manager. Avoid including secrets directly in your IaC code.

f. Document your code: Provide clear and concise documentation for your IaC code, outlining the purpose of each module, variables, and dependencies. Good documentation makes it easier for other team members to understand and maintain the code.

Real-world IaC examples

Now, let’s look at some real-world examples of IaC in action:

a. Deploying a web application: By using IaC tools like Terraform or AWS CloudFormation, you can automate the provisioning and management of web servers, load balancers, databases, and other necessary infrastructure components. This automation simplifies deployment and ensures a consistent setup across different environments (e.g., development, staging, and production).

b. Managing a multi-cloud environment: With IaC tools that support multiple cloud providers, such as Terraform, you can manage infrastructure across different clouds (e.g., AWS, Azure, and Google Cloud) using a single codebase. This approach streamlines multi-cloud management and ensures consistency across your infrastructure.

c. Scaling infrastructure on demand: By leveraging IaC and cloud-native services, you can automatically scale your infrastructure based on demand, such as increasing the number of web servers during peak traffic periods. This dynamic scaling ensures optimal resource utilization and improves overall application performance.

By implementing Infrastructure as Code best practices and exploring real-world examples, you can unlock the full potential of IaC in your organization. As you continue to refine your IaC skills, you’ll discover new ways to improve the efficiency and automation of your DevOps and sysadmin workflows.

Beginner's Guide to Infrastructure as Code (IaC) - Part 2: Popular IaC Tools and Getting Started

Sat, 15 Apr 2023 00:00:00 +0000

In the first part of our beginner’s guide to Infrastructure as Code, which you can find here we introduced the concept of IaC and its importance in the modern DevOps and sysadmin landscape.

In this second part, we’ll explore popular IaC tools and platforms and provide a hands-on guide for getting started with IaC in your organization.

Article Outline

Popular IaC tools and platforms
  a. Terraform
  b. Ansible
  c. Chef
  d. Puppet
  e. AWS CloudFormation
Getting started with IaC
  a. Choose the right IaC tool
  b. Learn the tool-specific language
  c. Create your first infrastructure code
  d. Test and validate your code
  e. Integrate IaC into your CI/CD pipeline

Part 2: Popular IaC Tools and Getting Started

Popular IaC tools and platforms

There are several IaC tools and platforms available, each with its own strengths and weaknesses. Here are some popular options:

a. Terraform: An open-source IaC tool developed by HashiCorp that allows you to define and manage infrastructure across multiple cloud providers using a declarative language called HCL (HashiCorp Configuration Language).

Example:

provider "aws" {
  region = "us-west-2"
}

resource "aws_instance" "example" {
  ami           = "ami-0c55b159cbfafe1f0"
  instance_type = "t2.micro"

  tags = {
    Name = "example-instance"
  }
}

b. Ansible: An open-source configuration management and automation tool that uses a simple, human-readable language called YAML. Ansible is agentless, relying on SSH for remote execution, and is often used for managing server configurations.

Example:

---
- name: Install and start Nginx on Ubuntu
  hosts: all
  become: yes
  tasks:
    - name: Install Nginx
      apt:
        name: nginx
        state: present

    - name: Start Nginx service
      service:
        name: nginx
        state: started

c. Chef: An open-source configuration management tool that uses a Ruby-based domain-specific language (DSL) to define infrastructure as code. Chef uses a client-server architecture and requires the installation of a Chef agent on managed nodes.

Example:

# cookbooks/my_web_server/recipes/default.rb

package 'httpd'

service 'httpd' do
  action [:enable, :start]
end

file '/var/www/html/index.html' do
  content '<html><body><h1>Hello, world!</h1></body></html>'
  mode '0644'
  owner 'root'
  group 'root'
end

d. Puppet: Another open-source configuration management tool that uses its own declarative DSL called Puppet DSL. Puppet can manage infrastructure in a client-server setup or in a standalone mode using Puppet Bolt.

class nginx {
  package { 'nginx':
    ensure => present,
  }

  service { 'nginx':
    ensure     => running,
    enable     => true,
    require    => Package['nginx'],
    subscribe  => File['/etc/nginx/nginx.conf'],
  }

  file { '/etc/nginx/nginx.conf':
    ensure  => file,
    owner   => 'root',
    group   => 'root',
    mode    => '0644',
    source  => 'puppet:///modules/nginx/nginx.conf',
  }
}

include nginx

e. AWS CloudFormation: A cloud-native IaC service provided by Amazon Web Services (AWS) that allows you to define and manage AWS resources using JSON or YAML templates.

Example:

Resources:
  MyS3Bucket:
    Type: AWS::S3::Bucket
    Properties:
      BucketName: my-example-bucket

Getting started with IaC

To get started with IaC, follow these steps:

a. Choose the right IaC tool: Based on your organization’s infrastructure needs, cloud provider(s), and existing skillset, select the most suitable IaC tool.

b. Learn the tool-specific language: Familiarize yourself with the syntax, structure, and best practices of the IaC tool’s language (e.g., HCL for Terraform, YAML for Ansible, or JSON for AWS CloudFormation).

c. Create your first infrastructure code: Start by defining a simple infrastructure component, such as a virtual machine or a network configuration, in your chosen IaC tool. Use the tool’s documentation and examples as a reference.

d. Test and validate your code: Ensure your IaC code is functional and adheres to best practices by using testing and validation techniques, such as static analysis, unit testing, and integration testing.

e. Integrate IaC into your CI/CD pipeline: Once you’re comfortable with your IaC tool and have successfully defined and tested infrastructure code, incorporate it into your CI/CD pipeline to automate the provisioning and management of infrastructure.

By following these steps, you’ll be on your way to successfully implementing Infrastructure as Code in your organization. As you gain experience and confidence with IaC, you’ll be able to further optimize your infrastructure management processes, reduce manual intervention, and increase the overall efficiency of your DevOps and sysadmin workflows.

Beginner's Guide to Infrastructure as Code (IaC) - Part 1: Introduction and Key Concepts

Fri, 14 Apr 2023 00:00:00 +0000

Introduction

Welcome to the first part of our beginner’s guide to Infrastructure as Code (IaC). In this series, we’ll be exploring the foundations of IaC, its benefits, and how to get started with implementing it in your DevOps and sysadmin workflows. By breaking down this topic into easily digestible sections, we aim to provide a comprehensive understanding that will help you leverage IaC for maximum efficiency.

Article Outline

What is Infrastructure as Code (IaC)?
Why is IaC important for DevOps and sysadmins?
Key concepts in IaC
Popular IaC tools and platforms
Getting started with IaC

Part 1: Introduction and Key Concepts

What is Infrastructure as Code (IaC)?

Infrastructure as Code (IaC) is the practice of managing and provisioning infrastructure components, such as virtual machines, networks, and storage, using machine-readable definition files rather than manual processes.

These definition files, or “code,” can be versioned, tested, and maintained just like any other software codebase. This approach allows for a more consistent, automated, and scalable infrastructure management process.

Why is IaC important for DevOps and sysadmins?

IaC has become a critical component of modern DevOps practices and sysadmin workflows for several reasons:

Automation: IaC enables the automation of infrastructure provisioning and management, reducing manual intervention and human errors.
Version control: With IaC, infrastructure configurations can be versioned and tracked, making it easier to maintain and update.
Collaboration: IaC promotes collaboration between teams, as infrastructure changes can be reviewed and approved by multiple stakeholders.
Scalability: IaC allows for infrastructure to be quickly and easily scaled to meet the demands of a growing application.
Cost savings: By automating and optimizing infrastructure management, IaC can lead to cost savings in the long run.

Key concepts in IaC

In order to understand and effectively work with IaC, it’s essential to be familiar with a few key concepts:

Configuration Management: The practice of maintaining and updating the state of your infrastructure over time.
Immutable Infrastructure: An infrastructure model where components are replaced with new instances rather than being updated in place.
Declarative vs. Imperative IaC: Declarative IaC focuses on defining the desired end state of the infrastructure, while imperative IaC outlines the steps to achieve that state.
Continuous Integration and Continuous Delivery (CI/CD): CI/CD practices enable the automation of building, testing, and deploying code and infrastructure changes.
Infrastructure as Code Testing: Techniques for validating and verifying IaC code to ensure the desired infrastructure state is achieved and maintained.

In the next part of our beginner’s guide to Infrastructure as Code, we’ll delve deeper into popular IaC tools and platforms, as well as provide a hands-on guide for getting started with IaC in your organization.

Stay tuned!

Configure Mikrotik with Cloudflare DDNS

Tue, 14 Feb 2023 00:00:00 +0000

On last year I published a guide to configure our Mikrotik with the OVH DDNS Service you can read the post in the next link: Configure Mikrotik with OVH DynDNS. A few days ago I migrated my DNS Zone from OVH to Cloudflare, and as you guessed I needed to change the DDNS from OVH to Cloudflare inside the Mikrotik script.

If you don’t know what DDNS is, it is a basic service where you have a local script or program to check when your public IP address changes and update the DNS record associated to it. This service is used by lots of people who have a homelab and are not given a static IP address from their ISP provider.

What I need

First of all, we need some information to use this script in your Mikrotik from Cloudflare:

DNS Zone in Cloudflare
Subdomain what we use to set the IP
API Token to configure the script
Script ddns_cloudflare for your mikrotik

After you create the DNS zone in Cloudflare and point the NS of your domain to it, create your custom sub-domain in your zone, like “private.mydomain.com” or something like that
To create your API token, refer to this article from Cloudflare from the Cloudflare docs
Download the ddns_cloudflare script

Gathering details for the script

The script needs some local variables to work properly. You need to get the zoneID, dnsRecordID, apiToken, email, subdomain from the DNS record and also the interface from Mikrotik where your public IP address is assigned to.

For the ZoneID you can find it on the Cloudflare dashboard. This is how to do it.
With your ZoneID, you can query your dnsRecordID via curl. This is API call. For example:

curl https://api.cloudflare.com/client/v4/zones/$ZONE_ID/dns_records \
    -H "Authorization: Bearer $CLOUDFLARE_API_TOKEN"

On the router, go and check what your WAN interface is. You can see that in the /ip address section.

Configure the script

In the script, variables are defined through the :local refs where you need to put the information:

:local cfzoneid "" // Cloudflare Zone ID
:local cfdnsrecordid "" // Cloudflare DNS Record ID
:local cftoken "" // Cloudflare API Token
:local cfemail "" // Cloudflare email user
:local cfdnshost "" // Cloudflare subdomain

:local publicinterface "" // Mikrotik Public interface

Just fill the spaces between double commas for each variable with the values you got in the previous step.

Setting up the DynDNS client script

Now go to your Mikrotik’s web interface and browse to the System -> Scripts menu, click on Add new, and fill in the form fields as follows:

- Name: ddns_cloudflare
- Policy: read, write, test

Now you can test the script by applying and clicking on Run Script. If everything is correct, you can see how on your Cloudflare’s dashboard the DNS record configured with the public IP address has been automatically updated. You also might like to use cli utilities such as dig or drill to check it out. That’s up to you :)

Scheduling the DDNS script

To achive this, from your Mikrotik’s web UI, go to System -> Scheduler menu, and click on Add new, then just fill each field with the next details:

- Name: ddns_cloudflare
- Interval: 00:10:00
- Policy: read, write, test
- On event: /system script run ddns_cloudflare

If you prefer setting this up through the cli, this is done just by running the following command:

/system scheduler
add interval=10m name=ddns_cloudflare on-event="/system script run ddns_cloudflare" policy=read,write,test start-time=startup

Remember that you can change the interval accordingly to fit your own needs.

With that, we can say we’re done! You have configured the Cloudflare DDNS on your Mikrotik, and now you can connect remotely to your home network by pointing your VPN configuration to your very own, automatically updated domain record.

See you next time, don’t forget to share & leave a comment!

Avoid private information leaks in your Docker images metadata

Mon, 13 Feb 2023 00:00:00 +0000

Please note that in this article we will not cover the deployment nor instructions on the usage of any version control system or container repository software, but are directly implying the use of both components.

If you are one of those who run their own servers at home or just away from being exposed to the Internet, but still get some of the work done there made public, you are probably interested in this article.

Recently I started publishing some of the Docker image builds I run on my own, for whoever might find the changes I introduce in others base images useful. Here is an example.

The issue

However, all the previous work before the image is pushed to public registries is run in my homelab. And for that reason I am especially interested in keeping all the details related to that part of my infrastructure private, as it is supposed to be.

This said, if you happen to run your own container registry but also want to make your images available (by uploading them to public container registries such as Docker Hub or GitHub Container Registry) there are a couple of things to be aware of.

In particular, I’m talking about these annotations: org.opencontainers.image.source and org.opencontainers.image.url (more details can be found here). When left intact, if you push a Docker image to a registry they will contain direct references to the Git repository where the image is being built from.

This is normally not a problem when the code is also hosted in public repositories, but this is not our case. Say our private Git server resolves to https://git.home.lab. Then we can check how it looks by inspecting the image, like this:

$ docker pull ghcr.io/mtnezm/nextcloud:latest
$ docker inspect ghcr.io/mtnezm/nextcloud:latest  |jq '.[] | .ContainerConfig | .Labels'
{
  "maintainer": "Miguel Martínez <mtnezm@linux.com>",
  "org.opencontainers.image.created": "2023-01-29T12:57:51Z",
  "org.opencontainers.image.description": "Custom Docker Nextcloud image build",
  "org.opencontainers.image.revision": "dab535e710a65affbb517aecaa9fc02fe21df1f6",
  "org.opencontainers.image.source": "https://git.home.lab/mtnezm/nextcloud.git",
  "org.opencontainers.image.url": "https://git.home.lab/mtnezm/nextcloud.git"
}

There it is, a link to our private Git repository included in a worldwide accessible Docker image. Something that does not make sense at all.

Fixing it

As we said, there is no real value in keeping a URL to a private Git server exposed there, so one thing we can do here is setting a custom value for those labels in our image. In this case, as I am using Drone CI to build them, it can be done by adding these lines to the corresponding step in the pipeline:

- name: "Deploy"
  image: plugins/docker
  (...)
  custom_labels:
    - org.opencontainers.image.source=REDACTED FOR PRIVACY
    - org.opencontainers.image.url=REDACTED FOR PRIVACY

If you are working directly with the Dockerfile, you can achieve it by adding these lines:

LABEL org.opencontainers.image.source=REDACTED FOR PRIVACY
LABEL org.opencontainers.image.url=REDACTED FOR PRIVACY

After applying that change, if we build, publish, pull and inspect the image again, we can verify how it looks now:

{
  "maintainer": "Miguel Martínez <mtnezm@linux.com>",
  "org.opencontainers.image.created": "2023-01-29T13:22:49Z",
  "org.opencontainers.image.description": "Custom Docker Nextcloud image build",
  "org.opencontainers.image.revision": "30f0f6cff0f9a63af0cd990301d1f301b425df85",
  "org.opencontainers.image.source": "REDACTED FOR PRIVACY",
  "org.opencontainers.image.url": "REDACTED FOR PRIVACY"
}

Conclusion

This way we can keep references to the work that runs in our homelab out of the public eye. But then again you might wonder “how do we know what is the work behind that image in particular, and if we can trust you?” which is perfectly reasonable.

In this case, as we are talking about Docker images there is the Dockerfile for you to go and verify it line by line, but let me answer to that one with a different approach in a future post here at Vectops.

Cheers!

Compile your own binaries for WD MyCloud OS5

Sat, 19 Feb 2022 00:00:00 +0000

Recently, I bought a WesternDigital MyCloud PR2100 NAS. It runs a simple Linux OS with some pre-built packages but somehow I feel it’s not enough and it could be used for more stuff, so I thought “can you imagine running Docker and deploying your own Dockerfiles on this thing?”. And well, now I have an answer for that: “yes, you can!”.

Introduction

Let’s see how it works. The NAS has a simple hardware setup:

Intel Pentium N3710 quad-core @ 1.60Ghz x86_64
4GiB DDR3 1600Mhz 2x2gb dual-channel
2 ports gigabit ethernet (bonding configurable)
2x SATA3 6Gbps ports hot-swap

This setup it’s already far from brand-new and has been present for a few years, mainly for non-technical audience. However, those are not bad specs at all. Just remember the latest Raspberry Pi v4 with 8GiB of RAM, for example. It’s not that far from it, right?

Anyway, you’re here because you want to install more packages in your WD MyCloud OS5, so let’s focus on that.

Grabbing the binaries

First of all, you need a computer with Docker engine already installed, say a laptop for example, then go ahead and clone this repository. It is where the WD Community puts its own source packages to be compiled locally and installed in your NAS:

$ git clone https://github.com/WDCommunity/wdpksrc.git
$ cd wdpksrc

Now, its time to let Docker work:

docker build -t wdpk .
docker run -it -v $(pwd):/wdpksrc wdpk /bin/bash

If everything went right, you will see some like this:

root@021ca29af42e:/wdpksrc# ls
Dockerfile  LICENSE  README.md	build.sh  build_and_install.sh	mksapkg-OS3  mksapkg-OS5  packages  tests  wdpk

On this environment, we can build any package inside wdpksrc directory. On this example, we are compiling a Docker package, so just run:

cd wdpksrc/docker
./build.sh

The process will take around 1 minute or so.

Installing packages on a WD NAS running OS5

After the process has been completed, you will have available a .tar.gz file containing the binaries on the packages/ directory. Uncompress the .tar.gz and install the package through your WD MyCloud OS5 web UI.

When you are done installing it, connect via SSH to your NAS. Then you should have available the docker command. Beside this installation, you will also have a Portainer web UI available on this URL: http://IP_NAS:9000/

That’s it! Now you can explore the public Docker images and run whatever you want in your NAS. If you are thinking of running a PiHole instance, just hold on and read the next post.

Cheers!

Dual booting Arch Linux and Windows

Fri, 18 Feb 2022 00:00:00 +0000

Assumptions

You are using systemd-boot as UEFI manager, NOT GRUB
You run Arch Linux and Windows on the same machine (it doesn’t matter if they are installed on different partitions or separate physical disks)

Copy EFI files

First things first! Fire up your Arch, get a root shell and mount the Window’s EFI partition:

mount /dev/sdXN /mnt # replace 'sdXN' accordingly to your Windows' disk address

Copy /mnt/EFI/Microsoft folder inside the /boot directory:

cp -r /mnt/EFI/Microsoft/ /boot/EFI/Microsoft/

Add Windows entry to systemd-boot

Add a new entry for Windows by creating a new /boot/loader/entries/windows.conf file, containing:

title   Windows 10
efi     /EFI/Microsoft/Boot/bootmgfw.efi

Apply changes by running:

bootctl update

Check that the new entry is showing correctly with:

bootctl list

It should output something like this:

Boot Loader Entries:
        title: Arch Linux
           id: arch.conf
       source: /boot/loader/entries/arch.conf
        linux: /vmlinuz-linux
       initrd: /initramfs-linux.img

        title: Windows 10
           id: windows.conf
       source: /boot/loader/entries/windows.conf

Once done, reboot your system and wait for systemd-boot to show up with both Arch and Windows 10 entries.

Until we meet again!

Configure Mikrotik with OVH DynDNS

Thu, 17 Feb 2022 00:00:00 +0000

Say you want to connect to your homelab or local network via VPN. This is possible thanks to software like WireGuard or ZeroTier among other older solutions (OpenVPN, PPTP, L2TP)…

Right, but you know that, in order to connect through this kind of software you need the public IP address of the VPN server. If you run this software in your home you probably have a dynamic public IP address assigned, as most of the ISP providers offer out there. So there’s a problem.

Introduction

Let’s pretend you can connect to your home’s VPN server no matter what IP address it has assigned, without paying an extra cent for it. Wouldn’t you love that? We do so too, and yep, there’s always a way.

The only thing you need to achieve this is your own domain (there are other solutions that don’t involve having a custom domain, but as those are non self-hosteable or fully manageable we won’t be covering them on this post).

With the DynDNS service you can update a type A DNS record when the record’s value changes, automatically, just by setting up a DynDNS client with a given timeout of your liking, say 5 minutes, 10 minutes, or 2 hours.

In this post we’ll explain how to configure your Mikrotik to use the OVH DynDNS feature. Before starting, please follow the link and configure your domain in OVH panel just as the guide says.

Pre-checks

Now, we also said this is for Mikrotik router users, so to configure OVH DynDNS in your Mikrotik, this is all we need:

- OVH DynDNS user
- OVH DynDNS password
- OVH DynDNS host (subdomain)
- Mikrotik Interface where is the public ip

To check your current’s Mikrotik public IP address, you can just issue this command via SSH or terminal web:

/ip address print

It will list all the address configured on your Mikrotik. Just look for the one marked with the D flag, that means Dynamic, and check the name of INTERFACE column. In case you have multiple dynamic addresses for whatever reason, look for the one matching the WAN interface.

Setting up the DynDNS client script

Now go to your Mikrotik web interface and browse to the System -> Scripts menu, click on Add new, and fill in the form fields as follows:

- Name: ovh-ddns
- Policy: read, write, test

For the source, you can just copy this script we already tested and tried.

After that click on OK. You need to configure the script, just review the first line and add the configuration we did get before:

:global ovhddnsuser "<OVH DynDNS USER>"
:global ovhddnspass "<OVH DynDNS PASS>"
:global theinterface "<INTERFACE THAT HAS YOUR PUBLIC IP>"
:global ovhddnshost "<OVHDynDNS HOSTNAME>"

Now you can test the script just by applying and clicking on Run Script. If everything is correct, you can see how, on your OVH panel, the DNS record configured with the public IP address has been automatically updated. Maybe you want to use a dig command to check it out. That’s up to you :)

Scheduling the DynDNS script

Right, but here’s the thing. At this point, if you don’t click on the “Run script” button, the script won’t trigger itself. Hopefully that is quite simple to fix: just configure a Scheduler to run this script on a given time basis (say 10 minutes).

To achive this, from your Mikrotik’s web UI, go to System -> Scheduler menu, and click on Add new, then just fill each field with the next details:

- Name: ovh-ddns
- Interval: 00:10:00
- Policy: read, write, test
- On event: /system script run ovh-ddns

If for some reason you prefer setting this up through the cli, just run this command via SSH:

/system scheduler 
add interval=10m name=ovh-ddns on-event="/system script run ovh-ddns" policy=read,write,test start-time=startup

Remember that you can change the interval accordingly to your fit your own needs.

With that, we can say we’re done! You have configured the OVH DynDNS on your Mikrotik, and now you can connect remotely to your home network by pointing your VPN configuration to your very own, automatically updated domain record.

See you next time, don’t forget to share & leave a comment!

Send multi-line Telegram messages using cURL

Thu, 17 Feb 2022 00:00:00 +0000

Not much context behind this one, I have been messing with stuff like this recently while doing some scripting and found it worth enough bringing it here. The solution is fairly simple, declare the needed variables:

BOT_API_TOKEN='<your bot token here>'
CHAT_ID='<your chat ID here>'
MSG='
This is a simple
multiline message
'

Then just build the requests like this:

curl --data "chat_id=${CHAT_ID}" --data-urlencode "text=${MSG}" 'https://api.telegram.org/bot'${BOT_API_TOKEN}'/sendMessage'
                                 ------------------------------

Stay safe!

3 ways to set up DynDNS with CloudFlare

Wed, 16 Feb 2022 00:00:00 +0000

Shout-out to all the selfhosters around there!

Ever thought about running your own dynamic DNS system without relying on third-party providers? Well, me too, not going to lie.

Luckily, if you own a domain and have a CloudFlare account, there’s a few things we can do about that. Today, we are bringing a bunch of different approaches to face this scenario, so let’s get down to it.

General assumptions

Have your own domain
Set up your domain’s DNS with CloudFlare
Have a CloudFlare account with DNS management capabilities
This task works with IPv4 only (for now)

Classic Linux cronjob with a simple cURL

Specific assumptions

Have access to a Linux host (no root permissions needed)
Have the jq package installed

Being this one the simplest way to achieve our goal, just save the following script to a file:

DNS_ZONE='<your DNS zone ID here>'
DNS_RECORD='<your DNS record ID here>'
AUTH_KEY='<your CloudFlare API token here>'
EMAIL_ADDRESS='<your CloudFlare's email account here>'
DNS_RECORD_NAME="\"<your.domain.tld here>\""
CURRENT_IP_ADDRESS="\"$(curl -s ip.me)\""
CURRENT_DNS_VALUE=$(curl -sX GET "https://api.cloudflare.com/client/v4/zones/${DNS_ZONE}/dns_records/${DNS_RECORD}" -H "Content-Type:application/json" -H "X-Auth-Key:${AUTH_KEY}" -H "X-Auth-Email:${EMAIL_ADDRESS}" | jq '.result["content"]')

if [ ${CURRENT_DNS_VALUE} != ${CURRENT_IP_ADDRESS} ]; then
	curl -sX PUT "https://api.cloudflare.com/client/v4/zones/${DNS_ZONE}/dns_records/${DNS_RECORD}" -H "X-Auth-Email:${EMAIL_ADDRESS}" -H "X-Auth-Key:${AUTH_KEY}" -H "Content-Type:application/json" --data '{"type":"A","name":'${DNS_RECORD_NAME}',"content":'${CURRENT_IP_ADDRESS}'}' > /dev/null
fi

Let’s say we name this file as cf-ddns.bash and save it to /opt. Then make it executable:

chmod +x /opt/cf-ddns.bash

And now let’s create a cron job that will run it every hour. Open the crontab editor with:

crontab -e

Then simply add the following line:

@hourly /bin/bash /opt/cf-ddns.bash

It is worth mentioning that the Linux machine where you are setting up this task must be behind the same NAT or on the same local network from where you want your DNS record to point to its public IP address.

Kubernetes cronjob

Specific assumptions:

Run a Kubernetes cluster (pretty obvious right?)

Why not!?

If you are running a self-hosted Kubernetes cluster you can just go all the way down and make use of the native cronjob feature it provides.

In this case you would only have to apply the following manifest, which creates a pod that will run the previous script we defined but it will be managed at a cluster level. (don’t forget to update the variable values according to yours!) :

apiVersion: batch/v1beta1
kind: CronJob
metadata:
  name: cf-dyndns
  namespace: default
spec:
  schedule: "@hourly"
  successfulJobsHistoryLimit: 0
  failedJobsHistoryLimit: 1
  jobTemplate:
    spec:
      template:
        spec:
          containers:
          - name: cf-dyndns
            image: bash:latest
            imagePullPolicy: IfNotPresent
            restartPolicy: OnFailure
            command:
            - /usr/local/bin/bash
            - -c
            - apk add --update curl jq &>/dev/null ; DNS_ZONE='<your DNS zone ID here>' DNS_RECORD='<your DNS record ID here>' AUTH_KEY='<your CloudFlare API token here>' EMAIL_ADDRESS='<your CloudFlare account's email here>' DNS_RECORD_NAME="\"<your.domain.tld here>\"" CURRENT_IP_ADDRESS="\"$(curl -s ip.me)\"" CURRENT_DNS_VALUE=$(curl -sX GET "https://api.cloudflare.com/client/v4/zones/${DNS_ZONE}/dns_records/${DNS_RECORD}" -H "Content-Type:application/json" -H "X-Auth-Key:${AUTH_KEY}" -H "X-Auth-Email:${EMAIL_ADDRESS}" | jq '.result["content"]'); if [ ${CURRENT_DNS_VALUE} != ${CURRENT_IP_ADDRESS} ] ; then curl -sX PUT "https://api.cloudflare.com/client/v4/zones/${DNS_ZONE}/dns_records/${DNS_RECORD}" -H "X-Auth-Email:${EMAIL_ADDRESS}" -H "X-Auth-Key:${AUTH_KEY}" -H "Content-Type:application/json" --data '{"type":"A","name":'${DNS_RECORD_NAME}',"content":'${CURRENT_IP_ADDRESS}'}' > /dev/null; fi && echo OK || echo ERROR

You can always check its status by running:

kubectl get cronjob

Drone CI pipeline

Specific assumptions:

Run a Drone CI instance
A git repository to interact with through Drone

Another way to automate your DynDNS setup could be defining the task into a CI pipeline, and for that purpose this time we will be using Drone CI to do so.

Although Drone provides a specific plugin to interact with CloudFlare’s API with ease, in this case we will be using a basic cURL image so we can get the DynDNS logic working in a pipeline inside our .drone.yml file.

There is a very specific reason for this: as we are dealing with dynamic IP addresses that may change over time, there are some limitations with Drone when it comes to sharing variable between steps in a pipeline.

Thus the workaround we are proposing here to bypass that limitation consists of having the ability to assign to the CURRENT_IP_ADDRESS variable a value at the very time the same pipeline’s step is running, not before.

In other words, not inheriting the contents from a pipeline-level variable but to gather its value from inside the same container that will also handle its content to update our DNS record itself on the next command it runs:

---
kind: pipeline
type: kubernetes # or "docker" if you don't have a kubernetes runner configured!
name: cf-ddns

steps:
- name: Update DNS record
  image: curlimages/curl
  environment:
    AUTH_KEY:
      from_secret: cloudflare_token
    EMAIL_ADDRESS:
      from_secret: cloudflare_email
    DNS_ZONE_ID:
      from_secret: cloudflare_zone_id
    DNS_RECORD_ID:
      from_secret: cloudflare_record_id
    DNS_RECORD_NAME: <your.domain.tld here>
  commands:
    - export CURRENT_IP_ADDRESS="$(curl -s ip.me)"
    - curl -sX PUT "https://api.cloudflare.com/client/v4/zones/$DNS_ZONE_ID/dns_records/$DNS_RECORD_ID" -H "X-Auth-Email:$EMAIL_ADDRESS" -H "X-Auth-Key:$AUTH_KEY" -H "Content-Type:application/json" --data "{\"type\":\"A\",\"name\":\"$DNS_RECORD_NAME\",\"content\":\"$CURRENT_IP_ADDRESS\"}

trigger:
  event:
  - cron

As you can see we are here using secrets to provide the multiple pipeline-level variable contents too, so don’t forget to create those secrets manually inside the repository options through the Drone web interface.

Once that’s done, the last thing we need is to create the cron job that will trigger the pipeline. The cron job options are easily found in the web UI as well, but if you’d rather using the cli you can always run something like this:

drone cron add "myuser/myrepo" "cf-ddns" '@hourly'

Worth to point out, this last method gets rid of the original principle: “if the IP address has changed, then overwrite the DNS record with the current IP address” that was present on the first two examples, and just runs the HTTP request to overwrite the DNS record with whatever IP address it gathers each time, no matter if it is a new one or not.

That’s all for now. Wish it was useful!

Cheers.

How to set up a SMTP relay on your Proxmox node

Tue, 15 Feb 2022 00:00:00 +0000

5-minute craft, specially useful for those self-hosted Proxmox users who seek sending e-mails to their remote inboxes directly from their nodes:

Assumptions

Your Proxmox run behind a NAT / is not directly accessible on the Internet so you find trouble sending native system e-mails by default
You have an already existing e-mail address you would want to make your Proxmox host use to send mails to you
The connection method to the “sender” e-mail address will be using password authentication

Step by step guide

First off, install some dependencies. Run as root:
```
apt-get update
apt-get install libsasl2-modules
```

Edit the /etc/postfix/main.cf file like this (replace relayhost’s key value to suit your needs):

# See /usr/share/postfix/main.cf.dist for a commented, more complete version

myhostname=pve.local

smtpd_banner = $myhostname ESMTP $mail_name (Debian/GNU)
biff = no

# appending .domain is the MUA's job.
append_dot_mydomain = no

# Uncomment the next line to generate "delayed mail" warnings
#delay_warning_time = 4h

alias_maps = hash:/etc/aliases
alias_database = hash:/etc/aliases
#mydestination = $myhostname, localhost.$mydomain, localhost
mynetworks = 127.0.0.0/8
inet_interfaces = loopback-only
recipient_delimiter = +

sender_canonical_classes = envelope_sender, header_sender
sender_canonical_maps =  regexp:/etc/postfix/sender_canonical_maps
smtp_header_checks = regexp:/etc/postfix/header_check

relayhost = <smtp.mailserver.com>:<smtp_server_port>
smtp_use_tls = yes
smtp_sasl_auth_enable = yes
smtp_sasl_security_options = noanonymous
smtp_tls_wrappermode = yes
smtp_tls_security_level = encrypt
smtp_sasl_password_maps = hash:/etc/postfix/sasl_passwd
smtp_tls_CAfile = /etc/ssl/certs/ca-certificates.crt

Edit or create the /etc/postfix/sasl_passwd file, that will contain the following line (remember to place your own values there):
```
<smtp.mailserver.com>:<smtp_server_port>    <sender>@<mailserver.com>:<password>
```
Change file permissions with:
```
chmod 600 /etc/postfix/sasl_passwd
```
Run:
```
postmap /etc/postfix/sasl_passwd
```
Edit or create the file /etc/postfix/sender_canonical_maps containing:
```
/.+/    <sender>@<mailserver.com>
```
Edit or create the file /etc/postfix/header_check with the following:
```
/From:.*/ REPLACE From: <sender>@<mailserver.com>
```
Apply changes by restarting Postfix service:
```
systemctl restart postfix.service
```

And that’s it. You can always manually test if it works by running:

echo "This is a test e-mail" | mail -s "Testing" <receiver>@<mailserver.com>

See you next time!

Terraform 1 and Proxmox; working as it should

Sat, 12 Jun 2021 00:00:00 +0000

And work it should…

A while ago we typed up a detailed article about using Terraform to provision VMs on proxmox nodes, you can find it here.

That article was dependant on some versions that are no longer supported and can and will cause problems eventually.

So we updated the dependencies, versions and did some definition refactoring in order to allow the process to work with the latest versions at the time of writing this article.

Current versions:

Terraform 1.0
Proxmox 6.4

Hooray Terraform Registry!

The Terraform Registry finally has Telmate’s Proxmox provisioner hosted on its platform so we can define it on the .tf file and forget about it.

You can find the module here: Telmate’s Proxmox Module

This allows us to happily say you no longer need to worry about installing the module manually and working out the kinks on Go’s installation methods.

Terraform Definitions

There are some changes that need to be made to the old main.tf definition file, the first one is:

terraform {
  required_providers {
    proxmox = {
      source = "Telmate/proxmox"
      version = "2.7.1"
    }
  }
}

Afterwards, we need change some other entries that are no longer supported:

Disk definitions

These definitions no longer apply as they are properly detected by the module:

    id              = 0
    storage_type    = "lvm"
    iothread        = true

Network definitions

The same happens to some of the network definitions:

    id              = 0

Github repo

We’ve uploaded the updates to Github with the updates you can also check out the commit history to see the changes yourself:

https://github.com/galdorork/tf-Proxmox

Also, we’ve updated the docker image we’ve use to run pipelines that need Terraform and (maybe) Terragrunt (wink, wink):

https://github.com/galdorork/terragrunt-proxmox-module

We’ll try to keep updating old posts to work on newer versions, but this one really needed some love.

BAIIII :)

How to fking use Git; like a pro

Fri, 11 Jun 2021 00:00:00 +0000

TLDR; Just read this:

https://git-scm.com/book/en/v2

Dear Devs,

Some tasks are easily performed using a GUI, some are not, some are more convenient. However, not all tooling is available all the time.

I recently had an issue with a version control interface that was failing, throwing 500 errors everywhere, but guess what? the git services were still working.

Its not only your responsibility, its your duty as a corporate developer to KNOW how to use git without all the bells and whistles of a graphical interface.

Using simple commands such as:

git branch
git checkout
git status
git merge
git commit --amend -m

Is extremely helpful and extremely important to know when you want to work fast.

Take this into account on your current/next job offerings.

PD: yes, this is a rant.

Build your own git server on WD MyCloud EX2

Sun, 04 Apr 2021 00:00:00 +0000

If you want to build your own server to create and manage a private repositories and if you have a WD MyCloud EX2, this post may be helpful to do it in a painless manner.

Fist of all, you need to enable the SSH access on your server, and install the Entware software.

After you have installed the Entware software, connect via SSH to your NAS and do the magic:

ln -s /opt/bin/opkg /usr/bin/opkg
opkg install git

Yes! Now you have installed the git software in your NAS.

Now we should make some syslinks to do git clone, upload, etc without errors, just execute these commands:

ln -s /opt/bin/git /usr/bin/git
ln -s /opt/bin/git-receive-pack /usr/bin/git-receive-pack
ln -s /opt/bin/git-shell /usr/bin/git-shell
ln -s /opt/bin/git-upload-archive /usr/bin/git-upload-archive
ln -s /opt/bin/git-upload-pack /usr/bin/git-upload-pack

Finally, you can use the WD MyCloud as a git server.

First steps with git server; the right way

Sun, 04 Apr 2021 00:00:00 +0000

Yay! What’s up? On this article we will summarize the first steps everyone should follow when it comes to deploying your own Git server. Don’t have any yet? Read this article.

Keep in mind that any host with the Git software installed and Internet connection can be a Git server, you just need to configure a SSH connection to be able to use it.

You can install Git on your Raspberry Pi, a virtual machine on your server, on the cloud… Whatever you want, just install it!

Remote server

Ok, right. Now you have the Git software installed, but you need to create the repository first. How to do it? Very simple, just go to the directory where you want to set up the repository (remember! It needs to be accessible through SSH):

# change directory
cd /home/cooluser/repositories/

# make new directory with .git extension
mkdir coolrepo.git

# come inside
cd coolrepo.git

# run the init command
git init --bare

Local machine

Create the directory where you want to have this repository

mkdir coolrepo
cd coolrepo

Now, you need to prepare this directory for git:

git init
touch README.md
git add .
git commit -m "First commit" -a
git remote add origin ssh://sshuser@server_ip:/home/cooluser/repositories/coolrepo.git
git push origin master

Et voila, you have your own repository with full functionality.

In the future, you can do a git clone like:

git clone sshuser@server_ip:/home/cooluser/repositories/coolrepo.git

Cheers!

Nagios-like monitoring Linux system services within Grafana

Mon, 29 Mar 2021 00:00:00 +0000

Assumptions

You have a Telegraf service up and running on the same host you want to monitor system services from
You have a InfluxDB instance up and running, receiving data from the previously mentioned Telegraf service
You have a Grafana instance up and running, making it possible to visualize data from InfluxDB
In this example, the Nginx web server will be monitored

Install Sysdweb

Follow the installation instructions provided in the repository’s README file.

Then save the sysdweb.conf file to /etc/sysdweb/sysdweb.conf:

sudo mkdir /etc/sysdweb
sudo wget https://raw.githubusercontent.com/ogarcia/sysdweb/master/sysdweb.conf -O /etc/sysdweb/sysdweb.conf

Create a Systemd service for Sysdweb

Save the following to /lib/systemd/system/sysdweb.service:

[Unit]
Description=Control systemd services through Web or REST API
Documentation=https://github.com/ogarcia/sysdweb
After=network.target
Requires=dbus.socket

[Service]
ExecStart=/usr/local/bin/sysdweb -c /etc/sysdweb/sysdweb.conf
Restart=on-failure

[Install]
WantedBy=multi-user.target

Start the new Sysdweb service:

sudo systemctl daemon-reload
sudo systemctl enable --now sysdweb

Check if the service is working:

$ curl -u 'sysdweb:supersecretpassword' 127.0.0.1:10080/api/v1/ngx/status
{"status": "active"}

Create a user for Sysdweb

There are multiple ways of interacting with Sysdweb (just review the config file), but this time I will be using the system user method:

sudo adduser sysdweb

Prepare Telegraf

Add the following block to /etc/telegraf/telegraf.conf:

[[inputs.http_response]]
  urls = ["http://localhost:10080/api/v1/ngx/status"]
  response_timeout = "5s"
  username = "sysdweb"
  password = "supersecretpassword"
  response_string_match = "{\"status\": \"active\"}"

Apply changes to the Telegraf configuration:

sudo systemctl restart telegraf

Let Grafana do the rest

After adding a new panel and selecting InfluxDB as data source, with such a simple query you will see a “Success” string (by picking Stat as visualization type) in your panel:

SELECT result_type FROM http_response WHERE server =~ /ngx/

At this point, by adding a value mapping from the string “success” to “UP”, you will then be able to monitor the current status of the Nginx service from your host at a glance.

In the same way, by adding another value mapping from the string “response_string_mismatch” to “KO”, it will become pretty clear that something wrong is taking place when Nginx goes down.

Final notes

I know there is a Telegraf plugin called inputs.systemd_units that could directly cover the need described in this article, but finding out that Sysdweb could be effectively used for the same purpose was hella fun.

Finally, shout out to Óscar García for bringing this useful tool to life!

Cheers!

Install Nextcloud with Apache2 on Debian 10

Sat, 30 Jan 2021 00:00:00 +0000

Pre-requisites

Have your own domain and be able to configure DNS accordingly
Have access to a Debian host with root privileges

Installing dependencies

Make sure to not miss this step:

apt update
apt install apache2 libapache2-mod-php php php-gd php-curl php-zip php-dom php-xml php-simplexml php-mbstring php-apcu php-mysql php-intl php-bcmath php-gmp php-imagick unzip mariadb-server certbot

Configure database

mysql -u root -p

CREATE DATABASE your_database;
GRANT ALL ON your_database.* TO 'your_user'@'localhost' IDENTIFIED BY 'your_password';
FLUSH PRIVILEGES;

Download and set up Nextcloud

Run the following commands:

cd /tmp
wget https://download.nextcloud.com/server/releases/latest.zip
unzip latest.zip
mv nextcloud/* /var/www/html/
mv nextcloud/.* /var/www/html/
rmdir nextcloud
chown -R www-data. /var/www/html/

Edit /var/www/html/config/config.php file and:

Declare your public access domain:

'trusted_domains' =>
  array (
    0 => 'your.domain.tld',
  ),

Disable new user registration:

'simpleSignUpLink.shown' => false,

Configure APCu as cache memory system:

'memcache.local' => '\\OC\\Memcache\\APCu',

Set up Apache2

Make it run at startup:

systemctl enable --now apache2

Enable HTTPS traffic:

a2enmod ssl

Issue a new Let’s Encrypt SSL certificate:

certbot certonly -d your.domain.tld

Set up Apache virtual host:

a2ensite your.domain.tld

Here’s a /etc/apache2/sites-available/your.domain.tld.conf file sample:

<VirtualHost *:80>
  ServerName your.domain.tld
  Redirect permanent "/" "https://your.domain.tld/"
</VirtualHost>

<VirtualHost *:443>
  ServerName your.domain.tld

  # Example SSL certificate path for Let's Encrypt
  SSLEngine On
  SSLCertificateFile /etc/letsencrypt/live/your.domain.tld/fullchain.pem
  SSLCertificateKeyFile /etc/letsencrypt/live/your.domain.tld/privkey.pem

  DocumentRoot /var/www/html

  CustomLog /var/log/apache2/your.domain.tld-access.log combined
  ErrorLog /var/log/apache2/your.domain.tld-error.log
</VirtualHost>

Apply changes by running:

apachectl configtest
systemctl reload apache2

At this point you should be able to open https://your.domain.tld at any web browser and follow the web installation wizard.

Fix security warnings on a fresh installation

Add HSTS header

Edit /etc/apache2/sites-available/your_vhost.conf and within HTTPS VirtualHost block add:

Header always set Strict-Transport-Security: "max-age=63072000; includeSubDomains; preload"

Increase default PHP memory_limit value

Edit /etc/php/7.3/apache2/php.ini and set:

memory_limit = 512M # At least

Disable PHP output_buffering

Edit /etc/php/7.3/apache2/php.ini and set:

output_buffering = off

Fix missing database indices

Run these commands:

chmod +x /var/www/html/occ
sudo -u www-data /usr/bin/php /var/www/html/occ db:add-missing-indices

Fix webdav URLs

Edit /etc/apache2/sites-available/your_vhost.conf and within the HTTPS VirtualHost add:

RewriteEngine On
RewriteRule ^/\.well-known/carddav https://your.domain.tld/remote.php/dav/ [R=301,L]
RewriteRule ^/\.well-known/caldav https://your.domain.tld/remote.php/dav/ [R=301,L]

Apply all these last changes by running:

systemctl restart apache2

Monitoring an OpenWRT router with Grafana and InfluxDB

Sat, 30 Jan 2021 00:00:00 +0000

Assumptions

You have:

A Debian 10 VM running InfluxDB (v1.7.10+) and Grafana (v6.7.2+) services
Network appliance running OpenWRT (v18.06+)

Install and configure InfluxDB (inside a VM)

Install needed package and dependencies:

sudo wget -qO- https://repos.influxdata.com/influxdb.key | sudo apt-key add -
echo "deb https://repos.influxdata.com/debian buster stable" | sudo tee /etc/apt/sources.list.d/influxdb.list
sudo apt update
sudo apt install -y influxdb

NOTE: if you receive the following error:

E: gnupg, gnupg2 and gnupg1 do not seem to be installed, but one of them is required for this operation

You need to install ‘gpg’ package:

sudo apt install gpg

Once InfluxDB is installed, edit /etc/influxdb/influxdb.conf file to enable Collectd plugin (locate and uncomment the following lines):

[[collectd]]
   enabled = true
   bind-address = ":25826"
   database = "${YOUR_DB_NAME}"
   retention-policy = ""
   typesdb = "/usr/local/share/collectd/types.db"
   security-level = "none"
   batch-size = 5000
   batch-pending = 10
   batch-timeout = "10s"
   read-buffer = 0

Create ’types.db’ file:

sudo mkdir -p /usr/local/share/collectd/
sudo wget -O /usr/local/share/collectd/types.db https://raw.githubusercontent.com/CactusProjects/openwrt_influxdb/master/types.db

Start and enable the InfluxDB service on startup:

sudo systemctl enable --now influxdb

Create Influx database:

influx
CREATE DATABASE ${YOUR_DB_NAME}
exit

Assign a one-month based retention policy to the database:

DROP RETENTION POLICY "autogen" ON "${YOUR_DB_NAME}"
CREATE RETENTION POLICY "one_month" ON "${YOUR_DB_NAME}" DURATION 730h0m REPLICATION 1 DEFAULT

Install and configure Grafana (inside a VM)

Install needed package:

sudo add-apt-repository "deb https://packages.grafana.com/oss/deb stable main"
sudo apt update
sudo apt install grafana

NOTE if you receive the following error:

bash: add-apt-repository: command not found

You need to install ‘software-properties-common’ package:

apt install software-properties-common

If you want to run Grafana service on port 80, edit the /etc/grafana/grafana.ini file:

http_port = 80

After that, you need to run this command to allow running Grafana service as non-root (the default user is ‘grafana’):

sudo setcap 'cap_net_bind_service=+ep' /usr/sbin/grafana-server

Start and enable the Grafana service on startup:

sudo systemctl enable --now grafana-server.service

Install and configure collectd (in OpenWRT)

Install needed packages:

sudo opkg install collectd collectd-mod-cpu collectd-mod-dns collectd-mod-interface collectd-mod-iwinfo collectd-mod-load collectd-mod-logfile collectd-mod-memory collectd-mod-network collectd-mod-openvpn collectd-mod-ping collectd-mod-rrdtool collectd-mod-thermal collectd-mod-uptime collectd-mod-wireless

Edit /etc/collectd/collectd.conf to configure the service:

BaseDir "/var/run/collectd"
Include "/etc/collectd/conf.d"
PIDFile "/var/run/collectd.pid"
PluginDir "/usr/lib/collectd"
TypesDB "/usr/share/collectd/types.db"
Interval 10
ReadThreads 2
Hostname "${YOUR_OPENWRT_HOSTNAME_HERE}"

LoadPlugin ping
<Plugin ping>
        TTL 127
        Interval 10
        Host "1.1.1.1"
</Plugin>

LoadPlugin memory
LoadPlugin cpu
LoadPlugin load
LoadPlugin uptime

LoadPlugin interface
<Plugin interface>
        IgnoreSelected false
        Interface "YOUR_INTERFACE_1_NAME_HERE" # i.e. "pppoe-wan"
        Interface "YOUR_INTERFACE_2_NAME_HERE" # i.e. "br-lan"
        Interface "YOUR_INTERFACE_3_NAME_HERE"
</Plugin>

LoadPlugin dns
<Plugin dns>
        Interface "YOUR_INTERFACE_1_NAME_HERE" # i.e. "pppoe-wan"
        Interface "YOUR_INTERFACE_2_NAME_HERE" # i.e. "br-lan"
        Interface "YOUR_INTERFACE_3_NAME_HERE"
        IgnoreSource "127.0.0.1"
</Plugin>

LoadPlugin thermal
<Plugin thermal>
        IgnoreSelected false
</Plugin>

LoadPlugin network
<Plugin network>
        Server "${YOUR_INFLUXDB_SERVER_ADDRESS_HERE}" "25826"
        CacheFlush 86400
        Forward false
</Plugin>

Start and enable Collectd service on startup:

sudo /etc/init.d/collectd start
sudo /etc/init.d/collectd enable

Testing it out

Once OpenWRT is configured, check if it’s working. SSH to your Grafana/InfluxDB server and issue:

influx
use ${YOUR_DB_NAME}
show measurements
select * from uptime_value

And verify if every 10 seconds the records are getting updated.

Grafana dashboard

CREATE NEW DATA SOURCE

Go to https://${YOUR_GRAFANA_SERVER_ADDRESS}/datasources and click on “Add”
Change “Name”
Fill “URL” field under HTTP settings
Fill “Database” field and set “HTTP method” to GET under “InfluxDB Details” settings

IMPORT AN EXISTING DASHBOARD

Place your mouse over the + sign at the left side of screen
Click on Import
In the Grafana.com Dashboard field, type “11858”
Click on the Load button

IMPORTANT!

There are things on that dashboard that won’t be working by default right after you create it, such as the Wi-Fi related stuff and maybe other elements depending on your setup.

At this point, it’s time for you to verify the queries definition and modifying it to suit your needs.

Keep your machines updated with GitLab; Ansible inside

Mon, 25 Jan 2021 00:00:00 +0000

Some of the more mundane tasks involving a server, or a bunch of servers, is to run repeatable tasks on each one of them.

Most sysadmins nowadays have some sort of automation in place to control this, however not everyone has the time to keep their machines updated, for example.

So, this article is going to be written as an absolute starting point to what can be achieved using pipeline scheduling and control for system automation.

How does a pipeline work in GitLab

According to GitLab’s documentation:

Pipelines are the top-level component of continuous integration, delivery, and deployment.

Pipelines comprise:

Jobs, which define what to do. For example, jobs that compile or test code.

Stages, which define when to run the jobs. For example, stages that run tests after stages that compile the code.

Jobs are executed by runners. Multiple jobs in the same stage are executed in parallel, if there are enough concurrent runners.

If all jobs in a stage succeed, the pipeline moves on to the next stage.

If any job in a stage fails, the next stage is not (usually) executed and the pipeline ends early.

So basically all we need to start automating things is a runner.

Luckily we’ve released an article that covers this, for example using the Kubernetes executor here.

Getting started with our repo

Since we’re using a pipeline, we’re also going to be needing a repository to save and commit our work.

Go on your GitLab instance and create a new repo. In this case we’re going to be calling it: vm-update

Once the repository has been created we need to create a folder on it, lets call it ansible:

mkdir ansible

Within that folder, we’re going to need a file called hosts and a folder called debian. Take into account that you can have several playbooks depending on the distro type, such as CentOS.

Actually, lets create a centos folder too. You should end up with the following files on the repo:

ansible/debian/
ansible/centos/

Ansible Hosts

For ansible to know where to connect to it needs a hosts file, create that file:

vi ansible/hosts

And add some machine descriptors to it (update with the IPs for your infra):

[debian]
120.0.120.200 # gitlab
120.0.120.201 # k3s
[centos]
120.0.120.202 # webserver

Ansible Playbooks

You’re going to need some playbooks for this to work.

Debian

Lets say you have a Debian type OS (Debian, Ubuntu, etc.) that needs to be updated, then create the following file:

vi ansible/debian/playbook.yml

Within that file place the following contents:

---
- hosts:  debian 

  tasks:

    - name: "Update repositories and upgrade packages"
      become: yes
      apt:
        update_cache: yes
        upgrade: yes
        force_apt_get: yes
        allow_unauthenticated: no
        autoremove: yes
        autoclean: yes
        install_recommends: no
        only_upgrade: yes
      tags: upgrade

CentOS

As for the CentOS based machines we can use this (more complete) playbook:

vi ansible/debian/playbook.yml

With the following contents

---
- hosts: centos

  tasks:
  
   - name: check packages for updates
      shell: yum list updates | awk 'f;/Updated Packages/{f=1;}' | awk '{ print $1 }'
      changed_when: updates.stdout_lines | length > 0
      args:
        warn: false
      register: updates
    - name: display count
      debug:
        msg: "Found {{ updates.stdout_lines | length }} packages to be updated:\n\n{{ updates.stdout }}"
    - when: updates.stdout_lines | length > 0
      block:
        - name: install updates using yum
          yum:
            name: "*"
            state: latest
        - name: install yum-utils
          package:
            name: yum-utils
        - name: check if reboot is required
          shell: needs-restarting -r
          failed_when: false
          register: reboot_required
          changed_when: false
    - when: updates.stdout_lines | length > 0 and reboot_required.rc != 0
      block:
        - name: reboot the server if required
          shell: sleep 3; reboot
          ignore_errors: true
          changed_when: false
          async: 1
          poll: 0
        - name: wait for server to come back after reboot
          wait_for_connection:
            timeout: 600
            delay: 20
          register: reboot_result
        - name: reboot time
          debug:
            msg: "The system rebooted in {{ reboot_result.elapsed }} seconds."

Now you should have the following files on your repo:

ansible/debian/playbook.yml
ansible/centos/playbook.yml
ansible/hosts

SSH Connections

For this to work the GitLab runner is going to need a key pair to be able to connect to the servers.

I’m not going to go on much detail about it on this article but you can create a folder on the repo to have those keys:

mkdir ssh

And put both the private and the public key there.

NOTE: I know this can be way more secure, lets save it for another article.

GitLab Pipeline

For a GitLab pipeline to work you’re going to need a .gitlab-ci.yml file:

vi .gitlab-ci.yml

And add the following contents:

image: mullnerz/ansible-playbook 

stages:
  - update_centos
  - update_debian

update_debian:
  stage: update_debian
  variables:
    ANSIBLE_HOST_KEY_CHECKING: "False"
    ANSIBLE_SSH_PRIVATE_KEY_FILE: "ssh/id_rsa"
  script:
    - chmod 600 ssh/id_rsa
    - ansible-playbook ansible/debian/playbook.yml -i ansible/hosts --tag upgrade -u vectops --private-key=ssh/id_rsa
  only:
    - master

update_centos:
  stage: update_cento
  variables:
    ANSIBLE_HOST_KEY_CHECKING: "False"
    ANSIBLE_SSH_PRIVATE_KEY_FILE: "ssh/id_rsa"
  script:
    - chmod 600 ssh/id_rsa
    - ansible-playbook ansible/debian/playbook.yml -i ansible/hosts --tag upgrade -u vectops --private-key=ssh/id_rsa
  only:
    - master

On this YML you can see we’re running a pre-setup Docker image that has all of the needed ansible tools to be used and the command that connects to the ansible hosts uses the vectops user.

Adjust it to your setup.

GitLab Scheduling

The whole idea is for this pipeline to run automatically on a scheduled day.

For this you can take advantage of GitLab’s job scheduler, on your GitLab web interface go on:

CI/CD > Schedules

Then click on the New Schedule and set up the properties and save the schedule.

Et voilà! You can now let the pipeline do it’s job and keep your machines updated.

Now, I know that some critical infrastructure can’t be updated this way because of some package updates can break stuff, however validation steps can be added between stages so the job is still automated but can be checked by a human before the upgrade happens.

Or maybe modify the pipeline to just update security patches that shouldn’t break anything.

This is just a starting point, it can be scaled or modified to suit your needs.

Kubernetes executor on GitLab (not a GitLab managed cluster)

Mon, 25 Jan 2021 00:00:00 +0000

Some of us that work with Gitlab, regardless on where we work or the role you thrive in (development, sysadmins, etc.), benefit hugely from GitLab’s ability to streamline and automate a deployment process.

Whether it’s a process that deploys an application or some tedious time-consuming process that sysadmins have to perform, can be automated.

Unfortunately not everyone or every company can afford to use the enterprise version of Gitlab. This is especially the case for Gitlab instances that run on homelabs.

Usually this doesn’t affect the functionality of the platform, however Gitlab has a really awesome toolset that allows it to manage a kubernetes’ deployment that’s included with the enterprise version.

We can work around this with the use of a Gitlab Kubernetes executor.

Enter the Kubernetes Executor

The Kubernetes executor is basically what it’s name states, it’s pod that’s deployed on a kubernetes namespace that can run processes.

What kind of processes? Anything that can be done with any other executor can be done by this executor.

Once deployed it’s a pretty hands-off experience. If you need to update it, you can update it with no downtime and the main benefit it has is that, since it’s not run on a specific VM or HW you don’t have another machine to add to your infrastructure and maintain. This last one can become overwhelming after a while (100 machines is easy to maintain, try 4000).

Requirements

This kind of executor needs some stuff to work:

A GitLab Instance
A Kubernetes instance, doesn’t matter which type, it can be a full fledged k8s cluster, OpenShift cluster or even k3s.
A namespace on the kubernetes cluster.
Access to create secrets on the namespace.
kubectl and helm3

That’s basically all you need to do this.

Setup

Kubernetes’ side

Namespace

You’re going to need a namespace to install the executor, in this case we’re going with gitlab-executor (make sure you can access the kubernetes instance from your machine):

kubectl create namespace gitlab-executor

GitLab registration Token

The registration token can be acquired from your GitLab instance, either from a full instance perspective, a GitLab-wide admin.

Or you can just define it for a project group or a single project.

The steps are pretty much the same.

For a project:

Go on the project settings -> CI/CD -> Runners -> Expand

And you’ll see the registration token.

Helm Chart

Once the namespace is created you need to set up Helm, start by adding the GitLab repo:

helm repo add gitlab https://charts.gitlab.io

Then update the repo:

helm repo update

Afterwards, you need a values.yaml file, which can be found here. From that file you’re going to need to focus on the following entries (to begin with):

gitlabUrl: https://gitlab.company.com/
runnerRegistrationToken: "XXXXXXXXXXXXXXXXX"
runners:
  image: ubuntu:16.04
  privileged: true

There’s a lot more entries on that file, you can decide which ones do you need, right now we’re focusing on the gitlabUrl, the registration token and whether the container is going to run on privileged mode (yes, this can run DockerInDocker).

Save the values.yaml on your local machine and then run the following command:

helm install gitlab-runner -f values.yaml -n gitlab-executor gitlab/gitlab-runner

Make sure you have the proper permissions to deploy an application on that namespace (although, if you’ve already created the namespace you should already have those.)

GitLab’s side

There’s not much to do on GitLab’s side, just wait until the runner is registered and you can use it.

Testing our shiny new runner

We’re going to perform an easy test,just a hello world to check that the runner runs a task.

For this you need a repo with the following files:

.gitlab-ci.yml

Yes, just one.

Within that gitlab-ci.yml file you need the following contents (we’re borrowing from the official GitLab wiki, the main article is here:

build-job:
  stage: build
  script:
    - echo "Hello, $GITLAB_USER_LOGIN!"

test-job1:
  stage: test
  script:
    - echo "This job tests something"

test-job2:
  stage: test
  script:
    - echo "This job tests something, but takes more time than test-job1."
    - echo "After the echo commands complete, it runs the sleep command for 20 seconds"
    - echo "which simulates a test that runs 20 seconds longer than test-job1"
    - sleep 20

deploy-prod:
  stage: deploy
  script:
    - echo "This job deploys something from the $CI_COMMIT_BRANCH branch."

After you commit and push the file to your project repo, the pipeline should run automatically and show the echo results.

Using Icinga2 and Ansible: one playbook to monitor them all!

Sat, 24 Oct 2020 00:00:00 +0000

Have you ever thought of way to monitor new hosts without having to spend much time adding the NRPE plugins, command check definitions and other custom configurations manually on each of them?

No problem, I have just faced that very same situation. Also, got tired of it pretty quickly. So how should we solve it?

The solution we are providing here is pretty simple: apply an Icinga2 monitoring template to a brand new, fresh installed machine thanks to Ansible.

NOTICE: for the examples provided we will be using Debian-like distros, so if yours is different you may have to adapt those affected parts, such as package manager related commands, specific Ansible plugins and so on.

Installing dependencies

The only things we need to configure on our machine are the SSH keys (so we can apply our playbooks normally), and to install the sudo package.

For the SSH keys you can copy your public key with the following command:

ssh-copy-id -i path/to/your/key ${YOUR_USERNAME}@${YOUR_NEW_MACHINE}

In case you don’t have a key set up, you can create one as follows:

ssh-keygen -t rsa

Then fill in the information the shell is going to prompt for. After that, from inside your new machine, run the following as root (or using sudo):

apt-get update
apt-get install sudo -y

When the package is installed, be sure to run visudo and configure the user you will be using properly, otherwise the Ansible steps may fail. If you are using root user directly (which I don’t recommend, insert security disclaimer here) these last steps are not needed at all.

Due to time constraints we’re not going to cover the Icinga2 installation on this article.

We’re going to assume you’ve already set it up and it’s running properly.

Setting up Ansible

From the machine you’re going to be using for the Ansible deployments, you will need to have a directory structure such as this one:

|-- inventories
|   `-- my_machines
|       `-- hosts
|-- playbooks
    |-- icinga_add_host.yml
    |-- install_nrpe_client.yml
    |-- files
        |-- nrpe
            |-- nrpe.cfg.template
            `-- nrpe_local.cfg.template

In this configuration, two playbooks are set up, the first one install_nrpe_client.yml has the following content:

---
- hosts: "{{ host }}"

  tasks:

    - name: "Install NRPE client and monitoring plugins"
      apt:
        pkg: ["nagios-nrpe-server", "monitoring-plugins", "nagios-plugins-contrib"]
        force_apt_get: yes
        update_cache: yes
        state: present
      tags: install

    - name: Copy NRPE service core files
      copy: src={{ item.src }} dest={{ item.dest }}
      with_items:
        - { src: 'nrpe/nrpe_local.cfg.template', dest: '/etc/nagios/nrpe_local.cfg' }
        - { src: 'nrpe/nrpe.cfg.template', dest: '/etc/nagios/nrpe.cfg' }
      tags: copy

    - name: Restart nagios-nrpe-server service
      service: name=nagios-nrpe-server state=restarted
      tags: restart

The playbook just goes on the machine and performs the needed package set up to run the monitoring services for that machine.

The second one, icinga_add_host.yml, has the following content:

---
- hosts: ${YOUR_ICINGA2_SERVER}

  tasks:

    - name: "Add host to Icinga"
      copy:
        dest: /etc/icinga2/conf.d/homelab/{{ host }}.conf
        content: |
          object Host "{{ host }}" {
            import "generic-host"
            address = "{{ host }}"
            vars.os = "Linux"
            vars.disks["disk /"] = {
              disk_partitions = "/"
            }
            vars.notification["mail"] = {
              groups = [ "icingaadmins" ]
            }
          }
      tags: add-host-template

    - name: "Restart Icinga2 service"
      service: name=icinga2 state=restarted
      tags: restart

Note that we’re using the default way of adding a host in Icinga2, of course this can be further extended by adding multiple and new commands and services.

Finally, there is the inventories/my_machines/hosts file, which should only have one line for now:

new_machine_hostname

Keep this in mind for later.

Setting up Icinga2

As you may have already seen, there are two other files in this setup, both templates are for the Icinga2 service configuration itself and command check definitions.

The file nrpe.cfg.template, is almost a clone of the default nrpe.cfg, as the only meaningful change to get things working is the allowed_hosts variable.

Where you must declare the address or FQDN of your Icinga2 server, so you can leave it intact except for that one bit (seriously don’t forget this).

The rest is just matter of custom preferences.

Also, the nrpe_local.cfg.template is the default file I chose to host all my custom command checks. However you can get things working too just by copy-pasting the ones declared in the default nrpe.cfg file or directly uncommenting them right there.

If you choose to copy them, it would end up looking something like this:

command[check_users]=/usr/lib/nagios/plugins/check_users -w 5 -c 10
command[check_load]=/usr/lib/nagios/plugins/check_load -r -w .15,.10,.05 -c .30,.25,.20
command[check_hda1]=/usr/lib/nagios/plugins/check_disk -w 20% -c 10% -p /dev/hda1
command[check_zombie_procs]=/usr/lib/nagios/plugins/check_procs -w 5 -c 10 -s Z
command[check_total_procs]=/usr/lib/nagios/plugins/check_procs -w 150 -c 200

### MISC SYSTEM METRICS ###
command[check_users]=/usr/lib/nagios/plugins/check_users $ARG1$
command[check_load]=/usr/lib/nagios/plugins/check_load $ARG1$
command[check_disk]=/usr/lib/nagios/plugins/check_disk $ARG1$
command[check_swap]=/usr/lib/nagios/plugins/check_swap $ARG1$
command[check_cpu_stats]=/usr/lib/nagios/plugins/check_cpu_stats.sh $ARG1$
command[check_mem]=/usr/lib/nagios/plugins/custom_check_mem -n $ARG1$

### GENERIC SERVICES ###
command[check_init_service]=sudo /usr/lib/nagios/plugins/check_init_service $ARG1$
command[check_services]=/usr/lib/nagios/plugins/check_services -p $ARG1$

### SYSTEM UPDATES ###
command[check_yum]=/usr/lib/nagios/plugins/check_yum
command[check_apt]=/usr/lib/nagios/plugins/check_apt

### PROCESSES ###
command[check_all_procs]=/usr/lib/nagios/plugins/custom_check_procs
command[check_procs]=/usr/lib/nagios/plugins/check_procs $ARG1$

### OPEN FILES ###
command[check_open_files]=/usr/lib/nagios/plugins/check_open_files.pl $ARG1$

### NETWORK CONNECTIONS ###
command[check_netstat]=/usr/lib/nagios/plugins/check_netstat.pl -p $ARG1$ $ARG2$

Running the playbooks

At this point everything should be ready to start monitoring your new machine, so the way you would do this is by running the following commands from your super amazing laptop:

ansible-playbook -i ansible/inventories/my_hosts/hosts ansible/playbooks/nrpe_client.yml --extra-vars "host=${new_machine_hostname}"
ansible-playbook -i ansible/inventories/my_hosts/hosts ansible/playbooks/icinga_add_host.yml --extra-vars "host=${new_machine_hostname}"

IMPORTANT: The “new_machine_hostname” value must coincide with the one set in the hosts inventory file (I told you to keep that in mind for a reason!), else it will return an error message.

Conclusion

As you can see, it can be pretty easy to monitor new hosts with Icinga2 when it comes to get things done by using automation software, and this is just a slight example.

Hope you find it useful!

Run your own bandwidth speed test server thanks to iPerf

Mon, 19 Oct 2020 00:00:00 +0000

When it comes to this kind of analysis, iPerf is one of the first options I think of to perform network tests.

Installing iPerf

So, that said let’s just get hands on. This example relies on running a Linux machine wherever you want to host your speed test instance. Thus the only thing you just need is to install the iperf3 tool.

Then the next step would be running the speed test instance, which can be achieved by running this:

/usr/bin/iperf3 -s -p 5500

Where -s means “run in server mode” and -p 5500 means “in port 5500” (this is the one by default, you can change it to whatever you want). If the server is running behind a NAT firewall don’t forget to set the proper port-forwarding rules.

Running the server

At this point, the only thing you have to do to start measuring your speed connection against this iPerf instance is running this command:

iperf3 -c ${IPERF\_SERVER\_ADDRESS} -p 5500 -P 8 -t 30

Where -c points the server to connect to, -P the number of parallel client streams to run and -t the ammount of time (in seconds) we want it to keep running.

You should get an output similar to this:

- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec  32.4 MBytes  27.2 Mbits/sec  444             sender
[  5]   0.00-10.00  sec  29.3 MBytes  24.6 Mbits/sec                  receiver
(...)
[SUM]   0.00-10.00  sec   276 MBytes   232 Mbits/sec  2663             sender
[SUM]   0.00-10.00  sec   259 MBytes   217 Mbits/sec                  receiver

Set it up as a Systemd service

Also, if you are running Systemd you can create a script file to make it run as a service on your system. For this, simply create the file `/etc/systemd/system/speedtest.service` and copy the following content into it:

[Unit]
Description=iPerf3 speed test server
After=network.target

[Service]
ExecStart=/usr/bin/iperf3 -s -p 5500

[Install]
WantedBy=multi-user.target

Once created just reload Systemd’s configuration and start it:

sudo systemctl daemon-reload sudo systemctl start speedtest.service

Lastly, if you wish to get your iPerf3 instance running on system startup go enable the new service:

sudo systemctl enable speedtest.service

And that’s it, now you can start measuring your bandwidth speed test against your own server.

See you next time!

Install Docker on Debian 10 with no effort

Sun, 21 Jun 2020 00:00:00 +0000

Recently, I have been dealing with the deployment of multiple Debian servers which I had to configure in a pretty tailored way and also running Docker was a must. As I was performing the deployments by hand, after a few times finishing the Docker service installation I reached the conclusion that things needed to speed up.

That said, the quickest way that I found to do it was as easy as parsing every command pointed in installation process (which you can check in the official Docker documentation) directly into the machines through a snippet that I posted on Gist to make it available for anyone interested on it.

So yeah, as you can guess this is another one-liner shot focused to save some time. So if you are thinking of installing Docker in a Debian system, just try:

curl -s https://code.nullbyte.es/docker-apt | sudo bash

Of course, I encourage you to check the content of whatever you run before doing so, for many reasons. In this case, note that by default I am installing the latest available version of docker-ce and also docker-compose.

When the installation of Docker finishes, you can check if it is active on your machine by running:

sudo systemctl status docker

After that, you should see something like this:

● docker.service - Docker Application Container Engine
   Loaded: loaded (/lib/systemd/system/docker.service; enabled; vendor preset: enabled)
   Active: active (running) since Thu 2020-06-20 15:42:47 CEST; 33s ago
     Docs: https://docs.docker.com
 Main PID: 3769 (dockerd)
    Tasks: 12
   Memory: 47.9M
   CGroup: /system.slice/docker.service
           └─3769 /usr/bin/dockerd -H fd:// --containerd=/run/containerd/containerd.sock

Note that there is also an official script from Docker to provide an automated installation located at this GitHub repository, so you just can follow any method you prefer, but keep in mind that, as the Docker team points, this method is only recommended in TESTING AND DEVELOPMENT ENVIRONMENTS.

And that’s all, now you can start containerizing whatever you want! I hope you find this as useful as I have. As always, any feedback is welcome.

Cheers!

Provision Proxmox VMs with Terraform, quick and easy

Thu, 07 May 2020 00:00:00 +0000

Update: We’ve uploaded an update to this article using the Terraform 1.0 release. You can find it here: Terraform 1 and Proxmox; working as it should

Previously, I wrote an article about how to provision Proxmox VMs using Ansible, you can find it here.

That article went into the workings of a functional Ansible script that provisions Proxmox virtual machines in an easy and streamlined way that can be integrated into many other implementations.

This time, we’re going to delve into another way to do so, using Terraform.

Terraform allows us to streamline the process even further by the use of plugins in much the same way as Ansible would.

Both of these methods depend on a template to be created beforehand. You could, of course, just create an empty virtual machine and it would work, but this means you’ll have to perform the OS installation manually, where’s the fun in that?

Within Proxmox, the VM creation method (using the GUI) is:

Create VM -> Present operating system ISO to VM -> perform installation -> Enjoy

That process takes too long, it’s manual (eww) and honestly, it’s just boring. Especially when you have to create multiple VMs.

Let’s create a template to be used by Terraform.

Building a template

Within Proxmox, you can choose two ways (usually) to create a VM, from the GUI or from the Terminal console. For this example, you should use a bare minimum machine with the minimal resources allocated to it. This way it can easily be scaled in the future.

You’re going to need a VM with the following resources:

1 Core 1 GB RAM 10 GB HDD 1 Network Interface 1 Cloud-init drive 1 EFI Disk

Some of the properties noted above will have to be added after the VM creation process.

Manually creating the template

The process for creating a VM within the Proxmox GUI has been explained countless times on the internet. For the sake of completeness let’s mention the basic process:

Click on create VM

Input a name for the VM, you can check for it to start at boot, your call. Click next
Select an ISO for the install and select the type and version of the OS that will be installed. Click next
Check the “Qemu Agent” option, you’ll use this later on. Click next
Select the Disk size, in this case 10 GB. You can also change some of the storage emulation options for this drive, we won’t go into that on this example. Click next
Select how many Cores you want to use for the VM, in this case 1 Core. Click next
Input the amount of memory for the VM, in this case 1024 MB. I advice using the Ballooning device so you can save up memory resources on the node and to be able to oversell the resources, just like a CPU. Note that if the memory is being actually used by the VM, it can’t be used by other VMs unless it’s the exact same memory block, enter KSM (Kernel Shared Memory) I won’t go into detail about KSM just know that it’s awesome. Select the Minimum memory for the Ballooning Device, in this case 256 MB. Click next
If you don’t have any custom network configurations on the node you can just Click next here. If you do, make sure that the configuration matches what you need.
Confirm the VM setup and click on “Finish”** Don’t start the VM yet**

After the VM is created, you’re going to need to change a few things on the VM. As you can see from the above steps, the Cloud-init drive wasn’t added. Select the VM on the left and click on Hardware then Add and finally on Cloud-Init Drive and select the storage where it will reside.

Afterward, edit the BIOS (double click on the BIOS entry on the Hardware tab) and select OVMF (UEFI)

Finally, the EFI Disk, it’s the same process as with the Cloudinit-drive, but now select EFI Disk and select the storage where it will reside. It won’t let you create the drive before setting up the BIOS on the second step.

Inside the VM’s terminal, you can go ahead and install the Cloud-Init’s packages so the VM is ready for use with:

apt-get install cloud-init -y

Using Debian’s official image for Cloud-init.

If the manual process above takes too long and you don’t want to spend as much time with the OS installation, you can just download a pre-configured image from Debian’s official repositories.

From the proxmox node’s terminal run:

wget https://cdimage.debian.org/cdimage/openstack/current-10/debian-10-openstack-amd64.qcow2

Since we described the process through the GUI on the manual installation, let’s go for the CLI way of doing things, the commands are as follows:

qm create 9000 -name debian-10-template -memory 1024 -net0 virtio,bridge=vmbr0 -cores 1 -sockets 1 -cpu cputype=kvm64 -description "Debian 10 cloud image" -kvm 1 -numa 1
qm importdisk 9000 debian-10-openstack-amd64.qcow2 lvm-thin
qm set 9000 -scsihw virtio-scsi-pci -virtio0 lvm-thin:vm-9000-disk-1
qm set 9000 -serial0 socket
qm set 9000 -boot c -bootdisk virtio0
qm set 9000 -agent 1
qm set 9000 -hotplug disk,network,usb,memory,cpu
qm set 9000 -vcpus 1
qm set 9000 -vga qxl
qm set 9000 -name debian-10-template
qm set 9000 -ide2 lvm-thin:cloudinit
qm set 9000 -sshkey /etc/pve/pub_keys/pub_key.pub

Please, please, please, take into account that the disk needs to be resized to 10 GB, so the VM has space to grow when it runs, you can do this from the hardware tab on the GUI.

Template setup

Ok, you created the template, now what?

The template needs to have some packages on it to run smoothly, not all of these packages are strictly necessary but it’s what I usually go for:

sudo apt install bmon screen ntpdate vim locate locales-all iotop atop curl libpam-systemd python-pip python-dev ifenslave vlan mysql-client sysstat snmpd sudo lynx rsync nfs-common tcpdump strace darkstat qemu-guest-agent

Defining the template

When the VM has been shut down cleanly, you can proceed and convert it to a template, this can be done on the Proxmox GUI by right-clicking on the VM and clicking on “Convert to Template”.

In this case, let’s rename the VM template to: debian-cloudinit

Success, the template has been created.

Enter Terraform

Terraform works in a pretty straightforward way.

It uses a file with HCL format (kinda like JSON) which is JSON compatible.

Within that file (or files), you can define an entire infrastructure. How simple or complex it can be is up to you. For this example, it’s going to be a pretty simple infrastructure definition, after all, we’re just creating one VM (for now).

The Terraform installation has been explained countless times online, for whichever operating system you might use, so I’m going to assume that you know how to install it (or how to google for: terraform install <insert OS here>).

Once it has been installed you need to install a provider so it can talk to the Proxmox API Server. Luckily there’s a provider that’s actively developed for this use.

Proxmox Provider

You can find the Proxmox provider for Terraform here.

The project is in active development and runs without hitches most of the time(99% of the time works all the time).

To install it just run the following commands to install the dependencies:

go get -v github.com/Telmate/terraform-provider-proxmox/cmd/terraform-provider-proxmox
go get -v github.com/Telmate/terraform-provider-proxmox/cmd/terraform-provisioner-proxmox
go install -v github.com/Telmate/terraform-provider-proxmox/cmd/terraform-provider-proxmox
go install -v github.com/Telmate/terraform-provider-proxmox/cmd/terraform-provisioner-proxmox
make

And finally, copy the executables that the compilation gave us into the path directory, in my case:

sudo cp $GOPATH/bin/terraform-provider-proxmox /usr/local/bin/
sudo cp $GOPATH/bin/terraform-provisioner-proxmox /usr/local/bin/

Terraform Project

Now you can get started with the Terraform project and project definitions. We’re going to use a directory structure like this one:

tfProxmox
|- main.tf

Just a single file. Remember this can be as complex or as simple as you need it to.

Project Definition

Within that main.tf file you first need to set up the connection profile for the Proxmox node. In case you have a cluster, any of the nodes will suffice:

provider "proxmox" {
    pm_api_url = "https://$PROXMOXSERVERIP:8006/api2/json"
    pm_user = "root@pam"
    pm_password = "$SUPERSECRETPASSWORD"
    pm_tls_insecure = "true"
}

Remember to change the $PROXMOXSERVERIP and the $SUPERSECRETPASSWORD variables on the example.

SSH Keys

Since you’re using a Cloud-init image (in case you went for Debian’s official template image), it’s set up for passwordless login so you need to define an SSH key to be installed on the VM:

variable "ssh_key" {
  default = "#INSERTSSHHPUBLICKEYHERE"
}

Where $INSERTSSHHPUBLICKEYHERE is your super-amazing-laptop’s SSH public key.

Now you can define the VM itself.

VM Definition

Bellow these definitions we can start defining our VM:

resource "proxmox_vm_qemu" "proxmox_vm" {
  count             = 1
  name              = "tf-vm-${count.index}"
  target_node       = "$NODETOBEDEPLOYED"
clone             = "debian-cloudinit"
os_type           = "cloud-init"
  cores             = 4
  sockets           = "1"
  cpu               = "host"
  memory            = 2048
  scsihw            = "virtio-scsi-pci"
  bootdisk          = "scsi0"
disk {
    id              = 0
    size            = 20
    type            = "scsi"
    storage         = "data2"
    storage_type    = "lvm"
    iothread        = true
  }
network {
    id              = 0
    model           = "virtio"
    bridge          = "vmbr0"
  }
lifecycle {
    ignore_changes  = [
      network,
    ]
  }
# Cloud Init Settings
  ipconfig0 = "ip=10.10.10.15${count.index + 1}/24,gw=10.10.10.1"
sshkeys = <<EOF
  ${var.ssh_key}
  EOF
}

Remember to change the $NODETOBEDEPLOYED entry for the node name where the VM will be deployed and the lvm-thin entry to whatever storage resource you’ll be using.

Let’s explain the resource definition. The main entries that you should take into account are:

count     <- This states the amount of VMs to be created
name      <- This states the VM Name, the "${count.index}" allows  
             you to create more than one VM and it'll just count    
             from then e.g.: tf-vm-1, tf-vm-2, tf-vm-3, etc.
cores     <- The amount of cores that the VM will have
memory    <- The amount of RAM the VM will have
disk      <- The disk definitions for the VM, scale the size here.
network   <- The network bridge definition to be used.
ipconfig0 <- The Ip for the VM, the "${count.index}" allows  
             you to create more than one VM and it'll just count    
             from then e.g.: 10.10.10.151, 10.10.10.152, etc.

Running Terraform

Terraform uses 3 main stages to run:

Init - This step allows Terraform to be initialized and downloads the required plugins to run
Plan - This step performs planning for the deployment, using the tf file that you’ve defined. It focuses on the calculation for the deployment and conflict resolution in case such conflict exists, it’s going to show you all the changes, additions, and deletions to be performed.
Apply - After the planning stage, this is the stage that applies the changes to the infrastructure. It’s going to give you a summary of the changes, additions and/or deletions to be made and ask for confirmation to commit these changes.

Init

While on the project folder run:

terraform init

As stated before, this is going to initialize Terraform and install the needed plugins for the project, the output should be as follows:

terraform init
Initializing the backend...
Initializing provider plugins...
Terraform has been successfully initialized!
You may now begin working with Terraform. Try running "terraform plan" to see
any changes that are required for your infrastructure. All Terraform commands
should now work.
If you ever set or change modules or backend configuration for Terraform,
rerun this command to reinitialize your working directory. If you forget, other
commands will detect it and remind you to do so if necessary.

Plan

This step will take care of all of the calculations that need to be run and conflict resolution with the infrastructure that might already be deployed.

victor@AMAZINGLAPTOP:~$ terraform plan
Refreshing Terraform state in-memory prior to plan...
The refreshed state will be used to calculate this plan, but will not be
persisted to local or remote state storage.
------------------------------------------------------------------------
An execution plan has been generated and is shown below.
Resource actions are indicated with the following symbols:
  + create
Terraform will perform the following actions:
# proxmox_vm_qemu.proxmox_vm[0] will be created
  + resource "proxmox_vm_qemu" "proxmox_vm" {
      + agent        = 0
      + balloon      = 0
      + boot         = "cdn"
      + bootdisk     = "scsi0"
      + clone        = "debian-cloudinit"
      + clone_wait   = 15
      + cores        = 4
      + cpu          = "host"
      + force_create = false
      + full_clone   = true
      + hotplug      = "network,disk,usb"
      + id           = (known after apply)
      + ipconfig0    = "ip=10.10.10.151/24,gw=10.10.10.1"
      + memory       = 2028
      + name         = "tf-vm-0"
      + numa         = false
      + onboot       = true
      + os_type      = "cloud-init"
      + preprovision = true
      + scsihw       = "virtio-scsi-pci"
      + sockets      = 1
      + ssh_host     = (known after apply)
      + ssh_port     = (known after apply)
      + sshkeys      = <<~EOT
              ssh-rsa ...
        EOT
      + target_node  = "pmx-01"
      + vcpus        = 0
      + vlan         = -1
+ disk {
          + backup       = false
          + cache        = "none"
          + format       = "raw"
          + id           = 0
          + iothread     = true
          + mbps         = 0
          + mbps_rd      = 0
          + mbps_rd_max  = 0
          + mbps_wr      = 0
          + mbps_wr_max  = 0
          + replicate    = false
          + size         = "20"
          + storage      = "data2"
          + storage_type = "lvm"
          + type         = "scsi"
        }
+ network {
          + bridge    = "vmbr0"
          + firewall  = false
          + id        = 0
          + link_down = false
          + model     = "virtio"
          + queues    = -1
          + rate      = -1
          + tag       = -1
        }
    }
Plan: 1 to add, 0 to change, 0 to destroy.
------------------------------------------------------------------------
This plan was saved to: planfile
To perform exactly these actions, run the following command to apply:
    terraform apply "planfile"

The plan states that a new resource will be created on the target node: pmx-01 (which is the node I’m using on my lab).

After you check the planning and everything seems to be alright apply it.

Apply

To apply the Terraform plan, just run:

terraform apply

This will give you the summary from the plan and prompt for confirmation, type: yes and it’ll do it’s bidding.

When it’s done the output should be as follows:

victor@AMAZINGLAPTOP:~$ terraform apply
...
...
...
yes
...
...
...
Apply complete! Resources: 1 added, 0 changed, 0 destroyed.
The state of your infrastructure has been saved to the path
below. This state is required to modify and destroy your
infrastructure, so keep it safe. To inspect the complete state
use the `terraform show` command.
State path: terraform.tfstate

Final Thoughts

This example should work as a starting point for Terraform interacting with Proxmox.

Take into account that you can destroy the VM by changing the count on the main.tf file to zero and going through the plan and apply stages.

Also, you can split the main.tf file into different files so it’s more organized when you decide to extend the infrastructure with different machine role definitions and different configurations for each one of them.

I’ve also uploaded the file on my GitHub here in case you just want the file.

Thanks for reading.

New Additions

As some folks have reported there could be some issues getting the proxmox plugin installed on their machines.

I’ve set up a Docker Image all set up and ready for it. The repo is located at:

https://github.com/galdorork/terragrunt-proxmox-provisioner

And here’s the link to the Image on Docker’s public registry:

https://hub.docker.com/r/galdor1/terragrunt-proxmox-provisioner

Setting up Python virtual environments: The right way

Thu, 16 Apr 2020 00:00:00 +0000

Yuuup little hackers, a few days ago I was assigned to a new project.

The main thing about this project is Python so it has to be set-up on my work laptop and the installation should work with my team’s laptops so they can help in writing some dependencies.

At this point, I have doubts about how my workload can shared with them, and the answer was easy, use virtualenvs.

First of all, you should know a little bit about pip.

Pip its a package installer for Python, its like apt, yum, pacman... but focused only to python dependencies.

You have probably heard about virtualenvs in Python, it allows you to create a Python environment totally independent of the local system where you can run specific versions of Python and its dependencies.

Install pipenv

Plain and simple; open your favorite terminal and issue this command:

$ pip install pipenv

Your virtualenvs packages will be saved at this location:

$HOME/.local/share/virtualenvs/

First environment with pipenv

Ok, let’s go to make our first environment. Just create a folder where you can put code or simply go to your git directory, and follow these steps:

$ mkdir myFirstApp
$ cd myFirstApp
$ pipenv shell

After you execute the last command, you will probably see a little change in your terminal. Yeap, your virtualenv has been created and you’re on it.

You have a new file called Pipfile, this file contains all the dependencies that pipenv has installed in your virtualenv as well as any dependencies that you project might need.

Let’s try to install, for example, the aws-cli tool:

(myFirstApp) $ pip install aws-cli

After the installation has finished, run help with help argument.

Now try to run an exit and test if you can to execute the aws-cli.

Activate your virtualenv again

$ source $HOME/.local/share/virtualenvs/myFirstApp-*/bin/active

Save your packages

This is a golden feature and this is the main thing we want to use.

I can install a lot of packages on my local machine, but when my teammate tries to deploy the application in his/hers local environment, there might be some dependencies issues. Mainly because they haven’t installed any.

How would this work?

Easy, just save your local packages on a stupid file called requirements.txt and push to your repository with the source code of the app.

$ pipenv lock -r | awk '{ if (NR!=1) print $1 }' > requirements.txt

After that, your teammate be able to install the dependencies with the command described on the next step.

Install packages from file

$ pip install -r requirements.txt

Aaaaand there go, you know now how to create and work with virtualenv, and the most important part: to be able to teamwork!

Production-ready Kubernetes PaaS in 10 steps; IaaS included

Tue, 25 Feb 2020 00:00:00 +0000

Easy, straightforward and can be performed by anyone with a basic understanding of production systems (read: basic). But, my advice IF (<- big if) you want to use these steps on your infrastructure, adapt them to your needs.

At the very least you’ll have a place to start without having to spend countless hours, doing research… drinking more than what anyone would consider a healthy amount of coffee. And/or having to work out what articles out there are compatible with what or which aren’t.

Tech writers need to start specifying software versions on their articles to prevent this from happening.

I’ve written other articles about how to streamline the deployment for VMs using virtualization platforms such as Proxmox and automation tools like Ansible.

This step-by-step can and will deal with some of these technologies without too much customization. You could also adapt it to use some other alternatives to the tools used here.

Enough preamble, let’s go for it.

Step 1: Get some hardware

Take your pick. Whether it’s a very low-cost decommissioned desktop PC from an office or a laptop with a broken screen.

You can get a lot of good deals on ebay for decommissioned hardware, either tower PCs or “old” servers.

Or a US$ 20,000 latest-generation server with the latest and greatest hardware.

It’s your call. I can’t determine how much load will your platform have but these steps should help you choose. So, please, adapt it to your needs.

Step 2: Let’s set up a Cloud-ish environment.

Let’s say you don’t have a lot of bare-metal machines or, maybe, you do. In my case I’ve had an issue with the bare-metal servers I have available to me. They’re too big.

Note: Yes, I know, this example can be set up with virsh. This is how I decided to do it, there’s a million ways to do it, this one’s easier for me :)

As in, 512 Gigs of RAM each. Now, Kubernetes’ workloads don’t usually go well with huge servers, there’s a pod limit to each node. Kernel limitations won’t help either. So I’ve decided to go with a Proxmox cluster environment that will host a few VMs with different VM models on it.

This is how the Proxmox deployment will look like after it’s done:

This example uses a 3 node Proxmox cluster. The setup for the cluster just needs you to install 3 machines with Proxmox, it can be Proxmox 5 or 6, your call. It makes no difference.

After you have installed the Proxmox distro on the machines, just create a cluster.

Create the Proxmox Cluster

In this example I’m going to reserve the following IPs for the installation (they’re also stated on the image above):

192.168.1.2 node1
192.168.1.3 node2
192.168.1.4 node3

The cluster creation is pretty well documented on Proxmox’s documentation here. On the newer versions, it can also be done with the GUI. I’m proceeding with the installation on Proxmox 6.1.

These steps should create a cluster pretty painlessly, let’s name it cloudish. Just log in via SSH to any of the nodes, in this case node1, and execute the following command:

node1:~# pvecm create cloudish

Then login into the other nodes and run the join command on each one of them:

node2:~# pvecm add 192.168.1.2
node3:~# pvecm add 192.168.1.2

The pvecm command will ask you for the node1 credentials, input them and you’re good to go.

Step 3: Create a bare-metal controller

To control and deploy the nodes I’m using MaaS (Metal as a Service), a bare metal controller that allows you to present the VMs you are going to create, or even bare metal servers to a cloud orchestrator such as Terraform or JuJu.

MaaS uses two elements:

A region Controller that handles: DNS, NTP, Syslog, and Squid Proxy
A rack Controller that handles: PXE, IPMI, NTP, TFTP, iSCSI, DHCP

Both of these elements can be deployed to the same machine.

In case you want/need high availability these components can be separated and installed in different machines, it’s well documented and straightforward, more info here.

In our case, we’re going to use the same machine for both things.

The deployment will look like this:

Download an Ubuntu Server ISO from the Ubuntu website and install it on a VM created on the Proxmox cluster, in this case, Ubuntu 18.04 LTS.

The bare minimum MaaS deployment needs one machine with 1 core and 2 Gigs of RAM. This should suffice for a very small deployment with a couple of machines.

Lets amp that up and go all the way to 4 cores and 8 Gigs of RAM!

For this example the MaaS machine will have the following IP:

MaaS --- 192.168.1.10

After you’ve installed the OS on the VM, go ahead and install the MaaS Controller, in our case MaaS 2.6.2:

maas:~# sudo add-apt-repository ppa:maas/2.6
maas:~# sudo apt update
maas:~# sudo apt install maas
maas:~# sudo maas init

After the initialization is done you’ll see a URL on the terminal. Go ahead and log into it using your favorite browser and finally, proceed with the configuration stage. Read more here.

Please remember to set up your DHCP service, by going on the MaaS interface and setting up the range you want.

To use MaaS DHCP you need it to be the only DHCP service on the network. This is very important. Also, reserve the ranges for the IPs you’re going to use elsewhere or the ones you’ve already used.

Step 4: Create VMs.

Yes, create Virtual Machines for Proxmox, a lot of them. I advise going with a for loop on the terminal using qm create so the process doesn’t get tedious, remember to check what requirements you need.

For this project, we’re going to create the bare minimum that Canonical recommends for a charmed Kubernetes deployment with HA, 10 machines.

Also the 3 VMs for JuJu.

The setup is as follows:

VM1    2 CPU Cores | 2 GB RAM | 20 GB HDD   tags=master
VM2    2 CPU Cores | 2 GB RAM | 20 GB HDD   tags=master
VM3    2 CPU Cores | 2 GB RAM | 20 GB HDD   tags=loadbalancer
VM4    2 CPU Cores | 2 GB RAM | 20 GB HDD   tags=easyrsa
VM5    2 CPU Cores | 2 GB RAM | 20 GB HDD   tags=etcd
VM6    2 CPU Cores | 2 GB RAM | 20 GB HDD   tags=etcd
VM7    2 CPU Cores | 2 GB RAM | 20 GB HDD   tags=etcd
VM8    4 CPU Cores | 4 GB RAM | 20 GB HDD   tags=worker
VM9    4 CPU Cores | 4 GB RAM | 20 GB HDD   tags=worker
VM10   4 CPU Cores | 4 GB RAM | 20 GB HDD   tags=worker
VM11   1 CPU Cores | 3 GB RAM | 20 GB HDD   tags=juju-controller
VM12   1 CPU Cores | 3 GB RAM | 20 GB HDD   tags=juju-controller
VM13   1 CPU Cores | 3 GB RAM | 20 GB HDD   tags=juju-controller

After you create the VMs you need to add them to MaaS, make sure they’re set up for Network boot first and HDD second, like this:

Then boot them up so they go through the process to be added to MaaS.

Step 5: Configure MaaS

For MaaS to be able to control Proxmox VMs like if they were a bare-metal machine it needs a driver.

Luckily a guy by the name of Wojtek Rakoniewski wrote one for all of us to use and posted it on launchpad. It works with what I’ve tested: Proxmox 5.1, 6.1 and everything in between since it uses the Proxmox API.

I’ve taken the liberty of hosting it on a public Github repo with credits for him all around, just in case the guys from launchpad decide to archive the bug tracker.

You can download it here.

Or using wget:

wget https://raw.githubusercontent.com/galdorork/proxmox-maas/master/proxmox.py

After you download it you need to put the file on this route on the MaaS Region controller:

/usr/lib/python3/dist-packages/provisioningserver/drivers/power/

And register the driver on the registry.py located here:

/usr/lib/python3/dist-packages/provisioningserver/drivers/power/registry.py

You need to edit the entry so it looks like this:

## Add this to the import headers
from provisioningserver.drivers.power.proxmox import ProxmoxPowerDriver
## The ProxmoxPowerDriver is the entry to be added
power_drivers = [
     ProxmoxPowerDriver(),
     ...
 ]
 ```
Then install the *proxmoxer* python package so the driver knows what to use when talking to the Proxmox API:
```bash
apt install python3-proxmoxer

Finally, restart the maas-regiond service to take the changes into account, systemctl restart maas-regiond should work.

Unfortunately even using this driver MaaS doesn’t know how to auto-detect the hardware controller for the machine, but it’s easily done manually on the VM’s config.

To do this you need to create a PVE user on the Proxmox GUI that has the VM.Audit and VM.PowerMgmt permissions.

Then configure the VM on MaaS so the region controller knows how to use it’s power capabilities.

Input here the PVE user you have created, it’s password, one of the proxmox nodes’ IPs and the VM id.

Step 6: Commission the VMs

MaaS uses with this workflow:

The machine is added →Machine needs to be commissioned → Machine is deployed.

The workflow makes sense since no machine can be formatted or deployed accidentally by being present on the same network and being pxe-booted.

When you boot up your proxmox VMs the first time, they will be added to MaaS as new machines:

Then you need to commission them, this means it gets booted from MaaS (remember to configure the power type) and some tests are run on the machine, as well as defining an inventory for that specific machine, parsing CPU info, RAM info, Storage info, etc.

After it’s commissioned it’ll be shown as ready:

This means the VM is ready to be used by either a controller or being manually deployed by either using the MaaS GUI or the MaaS API or, maybe even JuJu.

Let’s go for JuJu.

DO NOT FORGET: JuJu needs “tags” to identify MaaS nodes, else it grabs them at random, please set up tags with the VM usage so you can use them as constraints later on:

Step 7: Create your JuJu controller VMs.

This implementation requires a JuJu controller. For this example, we’re going to use `2.7.1-eoan-amd64.`

We’re going to use a single machine for it. The VM needs at least 1 core and 3 GB of RAM.

You don’t need to install it, just set it up to boot from Network and the VM will boot up using iPXE.

After it boots with the Ubuntu image it will download some of the packages it needs to check the resources the VM has and show it to the MaaS controller.

Afterward, commission it and you’re all set, you can commission your first machine from JuJu. This will be done from your PC.

On your PC, execute the following command:

yourawesomelaptop:~$ sudo snap install juju --classic

Now, add the MaaS cloud to your JuJu environment:

yourawesomelaptop:~$ juju add-cloud --local

The output will be as follows:

Cloud Types
  lxd
  maas
  manual
  openstack
  vsphere
Select cloud type: maas
Enter a name for your maas cloud: my-amazing-bare-metal-cloud
Enter the API endpoint url: http://192.168.1.10:5240/MAAS
Cloud "my-amazing-bare-metal-cloud" successfully added
You will need to add credentials for this cloud (juju add-credential my-amazing-bare-metal-cloud)
before creating a controller (juju bootstrap my-amazing-bare-metal-cloud).

Your JuJu installation needs to be able to connect to the API, it has the URL, now it needs the credentials:

yourawesomelaptop:~$ juju add-credential maas-cloud

It’ll ask for the MaaS secret, get it from the MaaS GUI. Click on your username on the top right corner and it’s the first field, copy and paste when it asks for maas-oauth:

Enter credential name: my-amazing-bare-metal-cloud-credentials
Using auth-type "oauth1".
Enter maas-oauth:
Credentials added for cloud my-amazing-bare-metal-cloud.

You can check the added credentials with:

yourawesomelaptop:~$ juju credentials --local

And finally create the main JuJu controller, using the JuJu tags and constraints:

yourawesomelaptop:~$ juju bootstrap --bootstrap-constraints tags=juju-controller my-amazing-bare-metal-cloud maas-controller

You’re done with JuJu for now.

If you want High Availability you can scale the controller and let it have more machines, 3 is advised for any production environment (remember to create the VMs first on Proxmox) for this example we’re doing it:

yourawesomelaptop:~$ juju enable-ha

Step 8: JuJu the hell out of that MaaS instance

Once the empty VMs are ready, the MaaS controllers are ready and the JuJu controller is ready you’re all set.

You can even do these steps on a nice graphical interface using JuJu Gui. Don’t worry, you can still use the terminal to adjust some of it, or even customize the charm bundle so it adapts to your machines.

In this case, we’re going to edit some of the constraints, so next to where it says `untitled-model` you can click on export and download the YAML descriptor file. On this file you need to focus on the machine descriptors at the end of the file and add the constraints to each one, depending on their need. E.g:

machines:
  '0':
    constraints: tags=easyrsa
  '1': 
    constraints: tags=etcd
  '2': 
    constraints: tags=etcd
  '3': 
    constraints: tags=etcd
  '4': 
    constraints: tags=loadbalancer
  '5': 
    constraints: tags=master
  '6': 
    constraints: tags=master
  '7': 
    constraints: tags=worker
  '8': 
    constraints: tags=worker
  '9': 
    constraints: tags=worker

Make sure the machine IDs match with the descriptions present at the beginning of each service definition. Then import it to your GUI and let it process.

Then commit the changes. You should add your `ssh` public key on this step. Remember you can always add it later on.

Then relax, get a cup of coffee, or tea, or a beer, maybe all of them. Depending on the hardware you have available this could take a while, from 15 mins on. In the meantime, you can see the deployment status on the GUI.

If you decide to use the terminal the bundle can also be deployed like this:

yourawesomelaptop:~$ juju deploy myawesomekubernetescluster.yaml

Step 9: Get your Kubernetes config file

After the provisioning is done you just need to get your Kubernetes config file from the juju deployment:

yourawesomelaptop:~$ juju scp kubernetes-master/0:config ~/.kube/config

And connect to the deployment using kubectl, in case you need to install kubectl on your machine I’ve written an article about it here, it has instructions for the main OSs out there.

There, you’re all set.

Optionally you can set up MetalLB as a way to expose the services or even use nfs-client provisioner or whichever persistent storage you might want to use.

Step 10: Scale it away

Yes, this platform can be scaled vertically, using more VMs present on MaaS for this task.

From JuJu you can extend each of the cluster roles from your choosing, an example for a Kubernetes worker would be:

juju add-unit kubernetes-worker

Scale it A LOT? (take caution, this will add 16 worker nodes):

juju add-unit kubernetes-worker -n 16

Or scale it down (removing the second worker):

juju remove-unit kubernetes-worker/2

The same applies to all of the other cluster roles.

Et voilá, it’s all set up, now you can focus on developing the next million-dollar idea without having to worry about the platform you have, and without paying premiums on cloud platforms that can be replicated on-site or maybe use hardware you already have laying around.

If you’re going to use this article as a base to deploy your own PaaS solution, don’t forget that it can be extended a lot, using JuJu for vertical scaling as well as Kubernetes autoscaling for horizontal scaling (pod replica scaling).

If you need any extra help you can get in touch with me using the comments on this article.

Force Apache to show a 403 error when accessing via IP address

Mon, 10 Feb 2020 00:00:00 +0000

Let’s say we have a few websites hosted in our machine, but we need them to process requests only via a name service; not when accessed via IP address. Very common behavior in shared hosting systems.

A real time saving tip, and it only needs a couple of lines of code. For this purpose we tested it on an Apache 2.4 server running under a CentOS system.

To achieve this we only need to make a new file (e. g. default.conf under the /etc/httpd/conf.d directory) containing the following lines:

<VirtualHost _default_:80>
   ServerName $WEBSERVER_IP_ADDRESS
   UseCanonicalName Off
   Redirect 403 /
</VirtualHost>

Of course we should run a reload to apply the changes we made on the service:

systemctl reload httpd

And that’s literally all, from here on out when anyone access the webserver through its IP address, the only thing that will show up is a beautiful 403 error message like this one (which is what we ever intended, right?) as long as you do not tune this up to show anything else or even a custom error message:

Forbidden

You don't have permission to access / on this server.

Cheers!

Provision Proxmox VMs with Ansible, quick and easy

Fri, 10 Jan 2020 00:00:00 +0000

Proxmox is an amazing Virtualization solution for both production, development, testing, etc. basically anything you can think of that requires a virtual machine.

Whether you rely on an actual kernel (full blown VM) or just the userspace (LXC Containers) it helps a lot to have a free tool that can perform. Not only that but the fact that it can be clustered and have VMs migrated from node to node can help with availability issues that can and will happen when you to perform maintenance on any specific virtualization node.

However, it can be cumbersome to provision VMs sometimes, the vanilla method is:

Create VM -> Present operating system ISO to VM -> perform installation -> Enjoy

This method can be time consuming depending on how many VMs you need, or what the OS installation process is like.

I know, I know… There’s new fancy tech such as Kubernetes that allows you to easily and swiftly deploy applications on a cloud environment but that kind of infrastructure is not always readily available and it can be hard to migrate some applications to it, depending on what has been developed.

Enter templates

The Proxmox system allows you to use and create VM templates, that can be set up with whatever operating system you want.

We’re going to use a basic Debian 10 template for this example, just go ahead and create a VM, pick low resources for the image so you can expand them later. CPU and memory are easily downsized, storage drives aren’t so take this into account.

I’ve created a VM with the following resources:

1 Core 1 GB RAM 10 GB HDD 1 Network Interface 1 Cloud-init drive 1 EFI Disk

Some of the properties noted above will have to be added after the VM creation process.

Creating the template manually

This process is pretty straightforward, here’s a step by step:

Click on create VM

Input a name for the VM, you can check for it to start at boot, your call. Click next
Select an ISO for the install and select the type and version of the OS that will be installed. Click next
Check the “Qemu Agent” option, you’ll use this later on. Click next
Select the Disk size, in this case 10 GB. You can also change some of the storage emulation options for this drive, we won’t go into that on this example. Click next
Select how many Cores you want to use for the VM, in this case 1 Core. Click next
Input the amount of memory for the VM, in this case 1024 MB. I advice using the Ballooning device so you can save up memory resources on the node and to be able to oversell the resources, just like a CPU. Note that if the memory is being actually used by the VM, it can’t be used by other VMs unless it’s the exact same memory block, enter KSM (Kernel Shared Memory) I won’t go into detail about KSM just know that it’s awesome. Select the Minimum memory for the Ballooning Device, in this case 256 MB. Click next
If you don’t have any custom network configurations on the node you can just Click next here. If you do, make sure that the configuration matches what you need.
Confirm the VM setup and click on “Finish” Don’t start the VM yet

After the VM is created, we need to add a couple of things.

After the creation

First, the Cloud-init drive, select the VM on the left and click on Hardware then Add and finally on Cloud Init Drive and select the storage where it will reside.

Second, edit the BIOS (double click on the BIOS entry on the Hardware tab) and select OVMF (UEFI)

Third, the EFI Disk, same process as the Cloudinit-drive, but now select EFI Disk and select the storage where it will reside. It won’t let you create the drive before setting up the BIOS on the second step.

Finally, start up the machine.

Lets get it prepared for Cloud-Init, go on the VM and run this command:

apt-get install cloud-init -y

That’s it, it’s set up now.

Using Debian’s official image for Cloud-init.

You can also just go for the easy step and more straightforward even that the manual installation, download a ready-to-go image from Debian’s repositories. SSH into the node, or open a shell on the node through the GUI and run:

wget http://cdimage.debian.org/cdimage/openstack/10.2.0/debian-10.2.0-openstack-amd64.qcow2

Afterwards create a VM either through the GUI or through the command line, if you decide to do it with the graphic interface just do as I wrote earlier, on the CLI the commands are as follows:

qm create 9000 - name debian-10-template - memory 1024 - net0 virtio,bridge=vmbr0 - cores 1 - sockets 1 - cpu cputype=kvm64 - description "Debian 10.2 cloud image" - kvm 1 - numa 1
qm importdisk 9000 debian-10.2.0-openstack-amd64.qcow2 lvm-thin
qm set 9000 - scsihw virtio-scsi-pci - virtio0 lvm-thin:vm-9000-disk-1
qm set 9000 - serial0 socket
qm set 9000 - boot c - bootdisk virtio0
qm set 9000 - agent 1
qm set 9000 - hotplug disk,network,usb,memory,cpu
qm set 9000 - vcpus 1
qm set 9000 - vga qxl
qm set 9000 - name debian-10-template
qm set 9000 - ide2 lvm-thin:cloudinit
qm set 9000 - sshkey /etc/pve/pub_keys/pub_key.pub

After you execute these commands you need to resize the disk to 10 GB, you can do this on the hardware tab for the VM

Installation

Start up the machine and install some basic packages you’ll most likely use on all the machines in my case I usually go for these:

sudo apt install bmon screen ntpdate vim locate locales-all iotop atop curl libpam-systemd python-pip python-dev ifenslave vlan mysql-client sysstat snmpd sudo lynx rsync nfs-common tcpdump strace darkstat qemu-guest-agent

After these packages are installed shutdown the VM with:

shutdown -h now

Defining the template

When the VM has been shutdown cleanly, you can proceed and convert it to a template, this can be done on the Proxmox GUI by right clicking on the VM and clicking on “Convert to Template”

Success, the template has been created.

Setting up the Proxmox node for ansible communication

The proxmox node has to be set up with a python tool called proxmoxer. This tool has the ability to behave like an ansible module, you can either run a playbook for the install or go in manually and proceed with the installation via SSH.

With a console proceed with these commands and install the necessary packages on the node:

apt install -y python-pip python-dev build-essential
pip update pip
pip install virtualenv
pip install proxmoxer

Creating our Ansible directory structure

We’re going to use a simple Ansible setup with the following structure:

hosts
playbooks
|──proxmox_deploy.yml
roles
|──proxmox_deploy
|
| ── defaults
| |
| ── main.yml
| ── meta
| |
| ── main.yml
| ── tasks
| |
| ── main.yml
| ── vars
| |
| ── main.yml
| ── travis.yml

Ansible files

You need to define several things on the ansible files, lets define the hosts file first.

For the sake of this example we’re using two nodes, proxmox1 with the IP: 192.168.1.11 and proxmox2 with the IP: 192.168.1.12

This definition only needs the proxmox nodes (in case you have more than one) and a group to be called:

hosts

[proxmox1]
proxmox1 ansible_ssh_host=192.168.1.11

[proxmox2]
proxmox2 ansible_ssh_host=192.168.1.12

[proxmoxs:children]
proxmox1
proxmox2

The main playbook is setup as follows (playbooks/proxmox_deploy.yml):

playbooks/proxmox_deploy.yml

- name: 'prep proxmox hosts for automation'
  hosts: 'proxmox1'
  vars_prompt:
  - name: PV_password
    prompt: "Node Password"
    private: yes
  - name: VM_name
    prompt: "VM name"
    private: no
  - name: VM_network
    prompt: "Network associated to ipconfig0"
    private: no
    default: vlan10
  - name: VM_IP
    prompt: "VM IP"
    private: no
    default: 192.168.1.100
  - name: VM_sockets
    prompt: "VM socket/s"
    private: no
    default: 1
  - name: VM_cores
    prompt: "VM core/s"
    private: no
    default: 1
  - name: VM_memory
    prompt: "VM RAM Memory (MB)"
    private: no
    default: 1024
  - name: VM_INCREASE_DISK
    prompt: "Increase virtio0 disk (20 GB) in"
    private: no
    default: 0
  - name: PV_node
    prompt: "Migrate Virtual Machine to"
    private: no
    default: none
  user: root
  gather_facts: false
  roles:
    - { role: proxmox_deploy, default_proxmox_node: proxmox1 }

This playbook defines the inputs that you, as a sysadmin/devops/computer-magician will need to input so the tasks can be completed successfully. Note: It asks for a “Node password”, this is so the proxmoxer python module can communicate with the node, it uses Linux standard PAM authentication

These inputs encompass CPU Sockets, CPU Cores, Memory, IP, Disk size and target node in case you want to migrate the VM to another node after it’s finished the creation process.

Then lets define the roles for this deployment, first with the travis.yml file within the roles directory (roles/proxmox_deploy/travis.yml):

roles/proxmox_deploy/travis.yml

---
language: python
python: "2.7"

# Use the new container infrastructure
sudo: false

# Install ansible
addons:
	 apt:
		 packages:
			 - python-pip

install:
	 # Install ansible
	 - pip install ansible

	# Check ansible version
	 - ansible - version

	# Create ansible.cfg with correct roles_path
	 - printf '[defaults]\nroles_path=../' >ansible.cfg

script:
	 # Basic role syntax check
	 - ansible-playbook tests/test.yml -i tests/inventory - syntax-check

notifications:
	 webhooks: https://galaxy.ansible.com/api/v1/notifications/

Afterwards we set up the main.yml within the defaults directory (roles/proxmox_deploy/defaults/main.yml). Please note that it’s very important to adjust this file to concur with the template name you created on the template creation step:

roles/proxmox_deploy/defaults/main.yml

---
# defaults file for proxmox_deploy
VM_template: debian-10-template
default_disk: virtio0
default_interface: ens18
default_volume: /dev/vda
default_partition: 2
template_name: template-debian-deployment

The handlers main.yml file is basically empty but needs to be defined (roles/proxmox_deploy/handlers/main.yml):

roles/proxmox_deploy/handlers/main.yml

---
# handlers file for proxmox_deploy

Then define a very basic default template for the meta’s main.yml file, i’m just leaving it as a default template (roles/proxmox_deploy/meta/main.yml):

roles/proxmox_deploy/meta/main.yml

galaxy_info:
	author: your name
	description: your description
	company: your company (optional)
	license: license (GPLv2, CC-BY, etc)
	min_ansible_version: 2.4
	galaxy_tags: []
dependencies: []

The main.yml file for the vars directory is as follows (roles/proxmox_deploy/vars/main.yml), in here, some of the variables that you might need for a VM are set up, in this case i’m going to use two VLAN setups as an example. Adjust it to your own infrastructure:

roles/proxmox_deploy/vars/main.yml

# vars file for proxmox_deploy
vlan10:
	params:
		netmask: 24
		vmbr: 0
		gateway: 192.168.2.1
		dnsservers: "192.168.2.253 192.168.2.254"
		searchdomain: vectops.com
vlan11:
	params:
		netmask: 24
		vmbr: 1
		gateway: 192.168.3.130
		dnsservers: "192.168.3.253 192.168.3.254"
		searchdomain: vectops.com

Finally the main file, the tasks’ main.yml file (roles/proxmox_deploy/tasks/main.yml). All the actual work goes here, the playbook uses this to complete all of the deployment tasks:

roles/proxmox_deploy/tasks/main.yml

---
# tasks file for proxmox_deploy
    - name: Cloning virtual machine from "{{ VM_template }}" with name "{{ VM_name }}" 
      proxmox_kvm:
        api_user : root@pam
        api_password: "{{ PV_password }}"
        api_host : "{{ default_proxmox_node }}"
        name : "{{ VM_name }}"
        node : "{{ default_proxmox_node }}"
        clone: "{{ VM_template }}"
        timeout: 300
      tags: provission,test

    - name: Increasing disk if it is necessary
      shell: A=$(qm list |grep "{{ VM_name }}" | awk '{print $1}'); qm resize $A {{ default_disk }} +{{ VM_INCREASE_DISK }}G
      when: '"{{ VM_INCREASE_DISK }}" != "0"'
      tags: provission

    - name: Waiting to apply cloud init changes in disk
      wait_for:
      timeout: 5
      tags: provission

    - name: starting new Virtual Machine to change IPv4 configuration, it is necessary
      proxmox_kvm:
        api_user : root@pam
        api_password: "{{ PV_password }}"
        api_host : "{{ default_proxmox_node }}"
        name : "{{ VM_name }}"
        node : "{{ default_proxmox_node }}"
        state : started
        timeout: 300
      when: '"{{ VM_INCREASE_DISK }}" != "0"'
      register: wait
      tags: provission

    - name: Waiting to start virtual server machine completely
      wait_for:
        timeout: 45
      when: wait.changed == true 
      tags: provission

    - name: Resize disk
      shell: growpart "{{ default_volume }}" "{{ default_partition }}"; pvresize "{{ default_volume }}""{{ default_partition }}"
      when: '"{{ VM_INCREASE_DISK }}" != "0"'
      delegate_to: "{{ template_name }}"
      tags: provission

    - name: stopping new Virtual Machine to change IPv4 configuration, it is necessary
      proxmox_kvm:
        api_user : root@pam
        api_password: "{{ PV_password }}"
        api_host : "{{ default_proxmox_node }}"
        name : "{{ VM_name }}"
        node : "{{ default_proxmox_node }}"
        state : stopped
        timeout: 300
      when: '"{{ VM_network }}" != "vlan10" or "{{ VM_INCREASE_DISK }}" != "0"'
      tags: provission

    - name: Loading set up for Virtual Machine. Assigning correct bridge in network interface
      shell: A=$(qm list |grep "{{ VM_name }}" | awk '{print $1}'); qm set $A - net0 'virtio,bridge=vmbr{{ item.value.vmbr }}'
      when: '"{{ VM_network }}" != "vlan10"'
      with_dict: "{{ vars[VM_network] }}"
      tags: provission

    - debug: 
        msg: "item.key {{ item.key }} item.value {{ item.value }} item.value.netmask {{ item.value.netmask }} item.value.vmbr {{ item.value.vmbr }}"
      with_dict: "{{ vars[VM_network] }}"
      tags: provission

    - name: Loading set up for Virtual Machine. Assigning IP, sockets, cores and memory for Virtual Machine
      shell: A=$(qm list |grep "{{ VM_name }}" | awk '{print $1}'); qm set $A - ipconfig0 'ip={{ VM_IP }}/{{ item.value.netmask }},gw={{ item.value.gateway }}' - nameserver '{{ item.value.dnsservers }}' - searchdomain '{{ item.value.searchdomain }}' - memory '{{ VM_memory }}' - sockets '{{ VM_sockets }}' - cores '{{ VM_cores }}'
      when: '"{{ VM_IP }}" != "automatic"'
      with_dict: "{{ vars[VM_network] }}"
      tags: provission

    - debug:
        var: current_ip
      tags: provission

    - name: Loading set up for Virtual Machine. Assigning IP automatically, sockets, cores and memory for Virtual Machine
      shell: A=$(qm list |grep "{{ VM_name }}" | awk '{print $1}'); qm set $A - ipconfig0 'ip={{ current_ip.stdout }}/{{ item.value.netmask }},gw={{ item.value.gateway }}' - nameserver '{{ item.value.dnsservers }}' - searchdomain '{{ item.value.searchdomain }}' - memory '{{ VM_memory }}' - sockets '{{ VM_sockets }}' - cores '{{ VM_cores }}'
      when: '"{{ VM_IP }}" == "automatic"'
      with_dict: "{{ vars[VM_network] }}"
      tags: provission

    - debug: 
        var: "{{ PV_node }}"

    - name: Migrating Virtual Machine if it is necessary
      shell: A=$(qm list |grep "{{ VM_name }}" | awk '{print $1}');qm migrate $A "{{ PV_node }}"
      when: '"{{ PV_node }}" != "none"'
      tags: provission

    - name: starting new Virtual Machine in current proxmox node
      proxmox_kvm:
        api_user : root@pam
        api_password: "{{ PV_password }}"
        api_host : "{{ default_proxmox_node }}"
        name : "{{ VM_name }}"
        node : "{{ default_proxmox_node }}"
        state : started
        timeout: 300
      when: '"{{ PV_node }}" == "none"'
      tags: provission

    - name: starting new Virtual Machine in correct proxmox node
      proxmox_kvm:
        api_user : root@pam
        api_password: "{{ PV_password }}"
        api_host : "{{ PV_node }}" 
        name : "{{ VM_name }}"
        node : "{{ PV_node }}"
        state : started
        timeout: 300
      delegate_to: "{{ PV_node }}"
      when: '"{{ PV_node }}" != "none"'
      tags: provission

This tasks file is pretty straightforward, it uses several steps, they’re defined as follows:

Clones the template into a new VM

If you choose to increase the disk size, it does so on the hardware side.
Applies Cloudinit configurations
Starts the VM and waits for it to start
Resizes the partition so it fits the new disk size (in case you did change it)
Stops the VM so it can apply the IP configuration
Assigns the correct network hardware properties, in case they need to be changed.
Configures the necessary hardware properties for the VM (cpu, memory, etc.)
If you chose to migrate it on the prompt, it will perform a migration of the VM to a target node

That’s it

That’s it, with this playbook you can easily deploy VMs on proxmox, fully configured to your needs with a simple ansible command:

ansible-playbook playbooks/proxmox_deploy.yml

I hope this helps you out as much as it has helped me to simplify and speed up the process of creating new classic virtual instances on Proxmox.

Backup your Docker containers, the lazy way

Wed, 08 Jan 2020 00:00:00 +0000

Disclaimer:

This method is valid for when you don’t have access to persist data from within the container to the host where it is running. To persist data on Docker containers it is recommended to rely on Docker volumes instead.

Ever wanted to save a snapshot of all your Docker containers but never had the time or even the energy to write a script that can do that for you? Don’t worry, we are here for that reason.

Let’s just say you save these snapshots under the /backup directory of your system. Then you just will run this magic one-liner:

for CONTAINER in $(docker ps -a --format '{{.Names}}'); do
	docker export $(docker ps -a |grep $CONTAINER \
	|awk '{print $1}') > /backup/backup_${CONTAINER}_$(date +%d-%m-%Y).tar
done

This will save the container’s current filesystem as a tarball.

However, remember that the docker export command DOES NOT include a copy of the contents of the volumes associated to the containers, as you can read in the official Docker documentation. For that purpose you will have to mess around with another kind of solutions such as file synchronization directly over the directories linked to the volumes or so.

Also, because of how it works, the docker export command will remove the entrypoint and the container’s history, making it useful only if you want to access the container’s information, but not if you look for running the image as a container again. The way you would access to it is first creating a new image from the tarball:

docker import /backup/backup_${CONTAINER}_$(date +%d-%m-%Y).tar ${IMAGE_NAME}

And then running a container based on the new image, like this:

docker run -ti ${IMAGE_ID} /bin/bash

If you are seeking to save a copy of a existing container keeping its information as it is, you would have to use the commit function in order to generate a new image based on the current container, like this:

docker commit ${CONTAINER} ${IMAGE_NAME}_$(date +%d-%m-%Y)

Then you can export it to a new tarball:

docker save > /backup/${IMAGE_NAME}_$(date +%d-%m-%Y).tar

So, if this is what you are looking to do with all your containers, you can run:

for CONTAINER in $(docker ps -a --format '{{.Names}}'); do
	docker commit $(docker ps -a |grep $CONTAINER \
	|awk '{print $1}') ${CONTAINER}_$(date +%d-%m-%Y)
	docker save > ${CONTAINER}_$(date +%d-%m-%Y)
done

And finally if you want to import your images from the tar files you can run:

docker load  < /backup/${IMAGE_NAME}_$(date +%d-%m-%Y).tar

See you next time!

How jq can save your job in simple ways

Wed, 04 Dec 2019 00:00:00 +0000

While checking the ticketing system for my company I noticed there was a new reply for a ticket that had been in hiatus for quite a while (hiatus as in it was opened and no one checked it for whatever reason, maybe someone was too busy and no it wasn’t on the backlog queue). The new message said something like: ‘Hey guys? is this happening? ever?’

So, I had some free time and decided to act on it. An API needed to be monitored. The API had several endpoints defined, all of which gave crucial information about the application instance that was deployed on the server. The information is presented on a JSON format and has the value entry defined with the values that needed monitoring.

Now, I’m not an expert on JSON, however the format isn’t too hard to understand and after checking some documentation and a couple of google searches it’s easily manageable. With a general understanding about the works on Bash, and a general understanding that’s basically summarized in two words -> google-fu (as 99% of sysadmins I know are too afraid to admit this publicly, I will).

Analizing

After checking the API’s URL with curl I could tell there were some endpoints presented. All of the endpoints the developers needed to be monitored in fact.

Note: If you don’t want an explanation for this and just wanna see the script, it’s posted at the end of this article

To do this all you need to use is a one-liner on the terminal such as this:

curl -s http://amazingapi.local/api

Note: you could use curl with any URL, be it with a domain name or an IP without DNS, if you need to get some more instructions about it feel free to consult the man pages for the command using man curl

Working with JSON

After checking the output from the curl command we get something like this:

curl -s http://amazingapi.local/api

{"names":["jvm.memory.max","jdbc.connections.active","process.files.max","jvm.gc.memory.promoted","tomcat.cache.hit","system.load.average.1m","tomcat.cache.access","jvm.memory.used","jvm.gc.max.data.size","jdbc.connections.max","jdbc.connections.min","jvm.gc.pause","jvm.memory.committed","http.server.requests","system.cpu.count","tomcat.global.sent","jvm.buffer.memory.used","tomcat.sessions.created","jvm.threads.daemon","system.cpu.usage","jvm.gc.memory.allocated","tomcat.global.request.max","hikaricp.connections.idle","hikaricp.connections.pending","tomcat.global.request","tomcat.sessions.expired","hikaricp.connections","jvm.threads.live","jvm.threads.peak","tomcat.global.received","hikaricp.connections.active","hikaricp.connections.creation","process.uptime","tomcat.sessions.rejected","process.cpu.usage","tomcat.threads.config.max","jvm.classes.loaded","hikaricp.connections.max","hikaricp.connections.min","jvm.classes.unloaded","tomcat.global.error","tomcat.sessions.active.current","tomcat.sessions.alive.max","jvm.gc.live.data.size","tomcat.servlet.request.max","hikaricp.connections.usage","tomcat.threads.current","tomcat.servlet.request","hikaricp.connections.timeout","process.files.open","jvm.buffer.count","jvm.buffer.total.capacity","tomcat.sessions.active.max","hikaricp.connections.acquire","tomcat.threads.busy","process.start.time","tomcat.servlet.error"]}

Not many people will read the entire output, first of all because even though it’s on a JSON format, it’s minimized so it’s not as easy to follow, however this is where our new best friend when it comes to JSON comes in: jq. If you pipe the output from the curl command into jq we can see something like this:

curl -s http://amazingapi.local/api | jq

{
  "names": [
    "jvm.memory.max",
    "http.server.requests",
    "jdbc.connections.active",
    "process.files.max",
    "jvm.gc.memory.promoted",
    "tomcat.cache.hit",
    "system.load.average.1m",
    "tomcat.cache.access",
    "jvm.memory.used",
    "jvm.gc.max.data.size",
    "jdbc.connections.max",
    "jdbc.connections.min",
    "jvm.gc.pause",
    "jvm.memory.committed",
    "system.cpu.count",
    "tomcat.global.sent",
    "jvm.buffer.memory.used",
    "tomcat.sessions.created",
    "jvm.threads.daemon",
    "system.cpu.usage",
    "jvm.gc.memory.allocated",
    "tomcat.global.request.max",
    "hikaricp.connections.idle",
    "hikaricp.connections.pending",
    "tomcat.global.request",
    "tomcat.sessions.expired",
    "hikaricp.connections",
    "jvm.threads.live",
    "jvm.threads.peak",
    "tomcat.global.received",
    "hikaricp.connections.active",
    "hikaricp.connections.creation",
    "process.uptime",
    "tomcat.sessions.rejected",
    "process.cpu.usage",
    "tomcat.threads.config.max",
    "jvm.classes.loaded",
    "hikaricp.connections.max",
    "hikaricp.connections.min",
    "jvm.classes.unloaded",
    "tomcat.global.error",
    "tomcat.sessions.active.current",
    "tomcat.sessions.alive.max",
    "jvm.gc.live.data.size",
    "tomcat.servlet.request.max",
    "hikaricp.connections.usage",
    "tomcat.threads.current",
    "tomcat.servlet.request",
    "hikaricp.connections.timeout",
    "process.files.open",
    "jvm.buffer.count",
    "jvm.buffer.total.capacity",
    "tomcat.sessions.active.max",
    "hikaricp.connections.acquire",
    "tomcat.threads.busy",
    "process.start.time",
    "tomcat.servlet.error"
  ]
}

If you need to install jq, it’s as simple as: apt install jq -y on Debian based distros or yum install jq -y on RHEL based distros.

This is A LOT easier to understand, and we start finding a small issue, there’s a lot of endpoints. There’s a few ways to create a script to monitor an API, most of them involve creating a single script for a single API, however there’s many APIs to be monitored in both this company’s infrastructure or whichever we want to explore.

Reasoning

In this specific case a few developers told me that not all of the APIs had the same endpoints but most of them didn’t need credentials since they were filtered by IPs, so it made the task a lot easier. I won’t go on details on how to do this against an authenticated API, maybe on another article; but implementing a token based authentication or a credentials based authentication on the script shouldn’t take more than a couple of lines.

Back to the API. As you and I can see, there’s a lot of endpoints and we don’t know if those are going to change any day, some deleted, some added. We need to make a script that can adapt and be set up within our monitorization tool to take into account the new changes. Also, this API is served from a Tomcat application, this doesn’t need to be the case always.

Since we can get the Endpoint list from the curl command we executed earlier, we can also check for a specific endpoint. In the case of the first entry: jvm.memory.max we can do this:

Selecting values

curl -s http://amazingapi.local/api/jvm.memory.max | jq

{
  "name": "jvm.memory.max",
  "description": "The maximum amount of memory in bytes that can be used for memory management",
  "baseUnit": "bytes",
  "measurements": [
    {
      "statistic": "VALUE",
      "value": 33096204287
    }
  ],
  "availableTags": [
    {
      "tag": "area",
      "values": [
        "heap",
        "nonheap"
      ]
    },
    {
      "tag": "id",
      "values": [
        "Compressed Class Space",
        "PS Survivor Space",
        "PS Old Gen",
        "Metaspace",
        "PS Eden Space",
        "Code Cache"
      ]
    }
  ]
}

Awesome, we see the value that the API provides as well as a few extra data that doesn’t serve much purpose for our needs right now and we won’t dwelve on it too much. The main attribute we want from the JSON output is the VALUE entry, which belongs on the measurements array:

  "measurements": [
    {
      "statistic": "VALUE",
      "value": 33096204287
    }
  ],

Options for jq

With jq we can get an array from within an object with single quotes and the brackets to denote an array, like this:

curl -s http://amazingapi.local/api/jvm.memory.max | jq '.measurements[]'

{
  "statistic": "VALUE",
  "value": 33096204287
}

In case we want to get a specific entry from the array we can go further in and get the specific value we need:

curl -s http://amazingapi.local/api/jvm.memory.max | jq '.measurements[]' | jq '.value'

33096204287

That number that outputs from the command is “The maximum amount of memory in bytes that can be used for memory management” as stated by the previous query we did against the API. This number can be converted a bunch of different ways from the script, however we won’t because we need the script to be as frugal as possible so it can be used with many APIs and many endpoints.

Now, we know how to get a specific value from an endpoint using curl, we just need to script it into something that can be used by monitoring software. Since most monitoring software allows bash scripts or sh or ksh or whichever shell is installed on the monitoring software’s host machine we’ll just use Bash.

Scripting

Please note that this script relies on some text manipulation that can be understood using the man pages on your linux console. This script might look kinda hard to read at first (sorry, not sorry) but it’s pretty simple once you actually go line through line.

Lets start with the first segment:

#!/bin/bash
if [[ -z $1 ]] ; then
    echo "This script has to be used with two parameters"
    echo "Usage:"
    echo "check_api.sh <api_url> <endpoint>"
    echo "e.g.:"
    echo "  check_api.sh http://exampleurl.com/api cpu.sys.util"
    exit 1
fi

We just initialize the script and put in a description, if the script doesn’t receive the parameters it expects on startup its going to show some help message and an example.

Afterwards we define a couple of variables to be used:

url=$1
probe=$2

endpoints=$`curl -s $1 | \
jq '.names[]'| awk -F\" '{print $2}'|cut -c 1-`

These variables do simple jobs. The first two are set up to receive the parameters from the script execution, the third one is the command that gets the enpoint list from the API server (like we discovered earlier).

Then we input the endpoint list onto an array defined as: endpointsarray:

endpointsarray=( $endpoints )

And finally the actual magic:

for i in "${endpointsarray[@]}"
do
        specs=`curl -s $1$i | \
        jq '.measurements[]' | jq '.value'`
        echo $i $specs |grep $probe | cut -d " " -f2-
done

This for loop just goes through the array and gets the value from the endpoint. It has some text manipulation due to the fact that the script is going to throw out some information that’s not needed, in case you wanna see some of it you’re free to experiment with the script, try seeing what the cut command does on it.

Hopefully this has thrown some light onto some things that may need some light. Full script as follows.

Final Script

TL;DR: Script that gets all endpoints from an API and gets the values from a single endpoint:

#!/bin/bash
if [[ -z $1 ]] ; then
    echo "This script has to be used with two parameters"
    echo "Usage:"
    echo "check_api.sh <api_url> <endpoint>"
    echo "e.g.:"
    echo "  check_api.sh http://exampleurl.com/api cpu.sys.util"
    exit 1
fi

url=$1
probe=$2

endpoints=$`curl -s $1 | \
jq '.names[]'| awk -F\" '{print $2}'|cut -c 1-`

endpointsarray=( $endpoints )

for i in "${endpointsarray[@]}"
do
        specs=`curl -s $1$i | \
        jq '.measurements[]' | jq '.value'`
        echo $i $specs |grep $probe | cut -d " " -f2-
done

Installing Arch Linux base (with UEFI inside!)

Wed, 04 Dec 2019 00:00:00 +0000

We’ve all strugled with installing Linux systems from scratch such as Arch or Gentoo, specially the first time we do so.

Up next is the most basic Arch Linux install you can achieve on a system without things breaking. While using an UEFI partition of course.

Requirements

Before anything starts you need to have an USB Drive with the Arch ISO on it. You can use whatever tool you want for this, including but not exclusively: Etcher, UNetbootin or even LiLi.

In case you’re doing this on a virtual machine, just have the ISO on hand as the virtualizer will need it.

Formatting

First off, boot the machine with the ISO or the USB drive.

You should take into account that this is going to format the drive on the machine you’re booting. I’ll say it again:

Caution, the following commands will erase the drive you’re using for the install, any data inside of them might be unrecoverable.

EFI Partition

After the machine has been booted we’ll be presented with a prompt, from there we can use gdisk to do the disk formatting. In this case you’re presented with the drive /dev/sda

gdisk /dev/sda

From gdisk delete any content that might be on the drive:

Create a new partition, in this case the EFI partition which will allow you to boot the machine with UEFI:

Press enter until you’re prompted with the size of the “Last Sector” and input the following:

+512M

Then select the type of filesystem that will be on the partition, since this is an EFI partition the code is:

EF00

Boot Partition

Afterwards we just create the main root partition for the system install:

To write these changes just press:

These steps will create two partitions:

/dev/sda1 -> EFI Partition
/dev/sda2 -> Root Partition

Filesystem

After the partition table has been created you can define the filesystems for the installation, for the EFI partition you need a vFAT filesystem, you can define it as follows:

mkfs.vfat /dev/sda1

As for the root partition, lets run things a little classic with ext4:

mkfs.ext4 /dev/sda2

Then mount them for the OS install, the /boot directory is created so the EFI has a place to be mounted on:

mount /dev/sda2 /mnt
mkdir /mnt/boot
mount /dev/sda1 /mnt/boot

Actual Installation

The fun part, you first need to select a mirror, the pacman repos have a lot of mirror worldwide, you can select the one that suits you the most, I won’t go into detail on how to decide since it depends on a lot of factors. Usually the one within your own country or the one closest to you should work best. You can find the repo list and edit it with:

vi /etc/pacman.d/mirrorlist

Strap it!, you can install the actual OS with the following command:

pacstrap /mnt base base-devel

This step will take a bit, depending on the type of drive you’re installing to, the speed of the machine and the speed of the USB drive, go and grab a cup of cofee on the meantime.

It’ll be done soon enough, you still need to install the bootloader for the machine to be able to boot into your new and shiny Arch installation.

Installing the bootloader

Installing the bootloader is pretty straightforward, in the case of Arch it’s kind of more hands-on than the other more user-friendly Linux distros such as ubuntu or other Arch derivatives such as Mandriva but it’s not as hard the old days… anyhow:

arch-chroot /mnt
bootctl install

And edit the bootloader’s configuration file with: vi /boot/loader/loader.conf and just leave the following entries (delete everything else):

default arch
timeout 4

For the final bootloader configuration we need to get the PARTUUID for the partition, do this with blkid and set it up on the entry for the Arch boot:

vi /boot/loader/entries/arch.conf

title Arch Linux
linux /vmlinuz-linux
initrd /initramfs-linux.img
options root=PARTUUID="YOUR_/dev/sda2_PARTUUID" rw

Again, YOUR_/dev/sda2_PARTUUID = The PARTUUID you got from the command: blkid

Finishing up

For the final step, and as a recommendation to avoid future lack of basic software components, just install some basic dependencies for the machine:

pacman -S linux linux-headers linux-firmware netctl dhcpcd mkinitcpio wpa_supplicant

Also, don’t forget to change the password, unless you want to watch how it boots up and you’re not even able to log in:

passwd

aaaaaaaand reboot:

reboot

There you go. You just installed a base Arch system, now go ahead and hack it up so it resembles whatever you want and whatever you want to use it for.

Good luck!

Kernel mishaps on the network stack

Wed, 04 Dec 2019 00:00:00 +0000

For a couple of months we’ve had some issues with the one of our servers, the issue was summarized as follows:

Dev: I can’t connect to the server right now

Ops: What do you mean?

Dev: Connections time out

Ops: I can connect, we can connect, everyone can connect. Stop messing with your OS settings.

Dev: …

Half an hour passes

Ops: Oh shit I can’t connect now.

Dev: :)

Now this incident happened a few times over the course of a few months, but since this was only happening from our offices and the server was located at the colocation site we thought it was something between the office and the datacenter. We could, however, connect to the server by jumping from any other server at the datacenter. This, of course, was not optimal but we managed to push the issue onto backlog since we had several other critical incidents to take into account.

After checking everything on the network we noticed something was off about the server. We couldn’t reproduce the issue on any server, including this one, totally intermitent and it didn’t point towards anything else other than the server.

Getting down to it

One day, I had some time to kill so I decided to take a kick at it. Note that this isn’t the best way to “kill time”, you’ll end up trying to kill the guys next to you after a few hours of nothing making sense.

So, I connected to the server via SSH, via mysql remote connection (port 3306), http, https, and a few others and couldn’t reproduce the issue. Checked it on a few other pc’s, Windows, Linux, Mac, a Solaris VM I carry around, nothing under the sun or above it could not connect. Wait it is.

After a few hours one of the computers failed to be able to connect, mine (praise the luck gods). So I started checking for eberything within that machine, connected via another machine on the Datacenter’s network.

Now, this machine had a few kernel parameters set up since it needed to use a few thousand connections every few hours due to some hadoop processes that were supposed to run on it, for more information check this article we wrote about it: here

Searching, searching, searching…

Then our almighty friend Google came in, then it’s less mighty friend Bing, then DuckDuckGo, then I was there looking at my screen with a few dozen tabs open with nothing to show for it and a light headache.

So I started looking at the networking stack within Linux and checking the network statistics recorded by the kernel itself. You can check these with:

netstat -s

The output for the netstat -s command left me with more questions than answers but something stood out of the terminal staring straight to my face:

35544811584 SYNS to LISTEN sockets dropped

This can happen since the Linux network stack is designed to avoid SYN floods by default and this counter is almost never zero because of false positives, however this doesn’t hinder the performance or the usability of the system. In case you want to learn more about SYN Flood attacks read here

The culprit

Other than the SYNS to LISTEN sockets dropped I noticed another couple entries:

7008728 invalid SYN cookies received
10703 packets collapsed in receive queue due to low socket buffer

These entries looked strange to me, I know this is a server but those numbers looked way too high for a server that wasn’t exposed to external networks.

After checking a lot more documentation than I should have about those entries which, by the way, is like reading a legal book with it’s own technical language that no one but kernel developers seem to understand. I decided to go back to the kernel config at:

/etc/sysctl.conf

And noticed this (this was copied verbatim straight from the server):

vm.overcommit_memory = 1
net.ipv4.tcp_fin_timeout=30
vm.swappiness=1
#net.core.somaxconn = 1024

#fs.file-max = 1311072
net.ipv4.tcp_max_syn_backlog = 16192
net.core.somaxconn = 4096
vm.nr_hugepages=512
net.ipv4.tcp_tw_recycle = 1
net.ipv4.tcp_tw_reuse = 1
net.netfilter.nf_conntrack_tcp_timeout_established=600

The entry for net.ipv4.tcp_tw_reuse = 1 was added by us on the article about reusing TCP ports but these other entries don’t match what I was expecting for this server to have configured. However, these sounded familiar and I decided to up the scale on some of them, including the tcp_max_syn_backlog and the somaxconn, adapting them to the memory quantities this server has, not outdated numbers for 4GB systems (this node has 512GB of RAM and averages 200GB free RAM, it can afford to give a few megs to the kernel)

But one other entry popped right at my eyes, after hours and hours of searching and watching for logs, and command outputs:

net.ipv4.tcp_tw_recycle = 1

Fixing the issue

Let me explain. tcp_tw_reuse and tcp_tw_recycle should be used when server and client timestamps are enabled, by default they are. tcp_tw_reuse only works for the client, tcp_tw_recycle works for both the client and the server, it reclaims the socket after 3.5*RTO (RTO=Retransmission timeout, if you want to read more about RTO, check this out) after opening the connection. In the case of LANs the retransmission is really fast since there’s basically no delay on the network so the server tries to recycle connections faster than they can be used, hence the no connection after a while.

Basically the server says “Nope, you already connected, I’m recycling your connection for something else”. But what else? …

The solution, after months, turns out to be disabling the entry. Just comment it out altogether from the sysctl.conf file:

net.ipv4.tcp_tw_recycle = 0

And reload it with a:

sysctl -p

Yeap, a few hours wasted, buuuuuuuuut. Now I know how the Linux network stack works, I’ll have a future article about it soon.

Recover an overwritten script from memory

Wed, 04 Dec 2019 00:00:00 +0000

Recently I had a co-worker do an oopsie on a server he was running a script on. He, actually wisely, executed a script on a screen within his terminal while connected to a production server. Now, this is all standard, what isn’t standard is that he somehow managed to overwrite the script (while in execution) on another terminal with a one-liner.

This script took a while to run so it was still running after he reported the issue to me. This is atleast 2 days of work from him, a guy that actually knows what he’s doing (unlike me :D ).

So I remembered from my old days that you could output a script from memory using GDB as debugger, so after a few man searches and a few hairs pulled out of my head (maybe a few from his head) and found a very useful command from within GDB to help us with this.

I can’t give you the actual script recovery, however I can replicate it.

Testing script

Assuming you have GDB we can replicate this procedure, if you don’t, you’re dead on the water. Just kidding, you can install it on whatever distro with the package gdb.

Let’s start with a very simple script within a screen:

#!/bin/bash

while true; do
    sleep 1; echo "Running..."
done

In the meantime we can leave the screen with Ctrl+A+D and then proceed to do some process debugging.

PIDs everywhere

Let’s start with getting the PID for the process:

ps -ef | grep bash

In this case we find the following output:

root       436   410  0 19:14 pts/0    00:00:00 /bin/bash ./while.sh

Debugging our way into the script

We want the PID number which is 436, so we attach ourselves to the process with our favorite open source debugger:

gdb -p 436

Now what we’re attached we enter the gdb prompt that will pause the process excecution while we’re attached, take this into account as it prevents a lot of stress thinking the process is going to end and lose everything we’re trying to rescue. Also, generate a core file:

(gdb) generate-core-file

Which will generate, you guessed it, a core file, with the output: Saved corefile core.436. This core file has everything in-memory regarding the process in execution, including our bash script. However we don’t want to suddenly stop the execution of the process so we detach from it using the command (insert obviousness here):

(gdb) detach

And we exit the debugger:

(gdb) quit

Analizing

This output is a binary file that isn’t stored as clear text, if we try to use cat to outpit it’s contents the system will say something like cat: write error: Input/output error so we use strings to make it usable as clear text (don’t have it? install it):

strings core.436 > rescuescript.sh

And within the dump we can find our script in clear text, however don’t be so hopeful and happy yet, core files can have a lot of data that isn’t useful for us, so be patient!. Anyway here’s the script we found on the core file (on line 10509, hence the need for patience):

#!/bin/bash
while true; do
    sleep 1; echo "Running..."
done

Et voilà!

Hope you don’t ever need to use this. Good luck if you do!

Retiring a Proxmox cluster node the right way

Wed, 04 Dec 2019 00:00:00 +0000

Disclaimer: Following the Official Proxmox documentation of course :p

When retiring a Proxmox cluster node from a cluster you need to take into account several things.

Not very often you find yourself with the opportunity to retire a server and install a new one. Especially a new, shiny one with all the best bells and whistles a provider can give. Or you can find yourself finding that a node has broken down and it’s not worth it to fix it, might as well replace it.

In case one of these machines belongs on a Proxmox Cluster you need to retire it. Leaving it presented on the cluster might cause a ton of issues we won’t go into on this article, trust me, Murphy’s law applies.

Is the proxmox node on?

If the Proxmox Node is online you need to migrate all the Virtual Machines (VMs) to another Node. Since Proxmox 5 migrating VMs has never been easier. You can just right click and click on Migrate, wether it has local or shared storage is transparent for the operator/sysadmin/you.

After the migration finishes (in the case that you don’t use shared storage and you use local storage this might take a while) you can proceed to shutoff the node.

Non-recoverable hardware failures on nodes don’t need to go through this step. There are ways to recover the VMs if the storage was locally based but, due to time constrains I won’t cover it here.

Make sure the node is no longer on

When the node has been shutdown DO NOT START IT UP AGAIN. This will cause problems with the corosync services. Preferably unplug the network connection, either physically or through the OS configuration.

Verify it!

In the case described on this post we’re using a 9 node cluster, something like this:

You need to check the Node name from the Proxmox Custer Manager CLI, this is as easy as opening an SSH console or going through the Proxmox GUI -> Shell and issuing the command:

pvecm nodes

It should give you an output like this:

Membership information
----------------------
    Nodeid      Votes Name
         9          1 proxmox9
         8          1 proxmox8
         3          1 proxmox3
         4          1 proxmox4
         7          1 proxmox7
         6          1 proxmox6
         5          1 proxmox5
         1          1 proxmox1 (local)
         2          1 proxmox2

Get down to it, and get the node down too

Let’s say we want to remove the node with id 2 (proxmox2). Ensure you’ve shut it off and issue the command:

pvecm delnode proxmox2

There won’t be any output to this command unless something goes wrong. What this command does is remove all of the configuration files from the pve file system that control the cluster. It does NOT, however, completely clean all of the corosync entries, hence the “If it’s out, it stays out”, the cluster will basically look like this afterwards:

Et voilá!

You’ve succesfully removed a Proxmox Node from a running cluster and with zero (hopefully) downtime on the VMs themselves.

Now go and add a brand new shiny machine to the Cluster. Maybe a few more than one!

If you want to check out how to create a Production-Ready Proxmox Cluster or maybe adapt your current deployment to the different approaches we’ve explored deploying Production Proxmox clusters between all of us here at Vectops you might want to check the upcoming series about it.

Reusing TCP ports for fun, profit and lack of headaches

Wed, 04 Dec 2019 00:00:00 +0000

Last week I was checking out some issues on the ticketting system that hadn’t been solved way before my time. While browsing them I noticed an unsolved ticket that had a lot of updates from both the sysadmins that came before me and the developers from an application that ran a few times per day but always had issues running.

The errors that were thrown by the application (which was programmed in Java of all things) were like this:

java.net.NoRouteToHostException: Can't assign requested address (Address not available)

Possible causes

All of the updates regarding this issue “pointed” towards the database cluster that was being called in order for the application to store it’s output (the quotes serve a purpose by the way, just wait and read). Now, I could see a year’s worth of interactions from both teams discussing the error and possible solutions but somehow they all managed to “point” the error towards the database cluster.

Please take into account that this is a Percona cluster that could handle tens of thousands of connections, not counting the concurrency. Real beefy stuff. That had been configured by a co-worker which I know can do his job competently, but since the ticket had my attention I decided to take a crack at it.

TroubleShooting

The first few steps of the troubleshooting I had to do were to eliminate all obvious conclusions (year’s worth of work, remember?) my co-workers came to; which pointed me towards the Percona cluster. Since this is a production system I wasn’t able to really stress it since I knew my testing would get unwanted attention from developers and managers that start yelling wolf as soon as they see something that’s not expected on their metrics. So I just threw arround 30k concurrent sessions to the cluster and it didn’t break a sweat.

So, next steps it is. After checking out the whole error log from Java I noticed the error and decided to SSH into the application server from where the app was being deployed and as soon as I connected to it I received a Slack message asking me not to play with since it was on production!!!

Naturally I did what every sysadmin does atleast once (a day), ignore the developer and started checking out every log I could get my hands on and noticed something specific to those machines, they weren’t reusing TCP connections. As you can see it didn’t “point” to a database issue. These kinds of database clusters are designed to run with HUGE workloads.

Possible solutions

At this point it was painfully obvious that the machine was running out of internal ports for the application that was deployed on it, in order to check this I just reviewed the kernel parameters in execution with the following command:

sysctl -a | grep range

And the reusability of the ports:

sysctl -a | grep reuse

Definite solution

Both of those commands gave me what I needed, the port range was ok:

net.ipv4.ip_local_port_range = 15000    65000

Which is expected since it’s a default behaviour, but the reuse flag wasn’t enabled, so we just changed it on the kernel config on the file:

/etc/sysctl.conf

And added the entry:

net.ipv4.tcp_tw_reuse = 1

Applied it live:

sysctl -p

And, done.

Et voilà!

Final thoughts

As soon as I had finished the changes, I went to the developers desk and described my solution and the horror face he made when I told him I needed to change a kernel parameter was priceless. He kept rambling about how it needed to be scheduled since it needed a reboot and this is a production system that had to approved by the higher-ups and bla bla bla bla bla (Good thing those types of changes don’t require a reboot). I just told him it was done and added a simple “Please run the application again, it won’t fail now”.

Hope it doesn’t :p

S(z)etting up your private networks with ZeroTier

Wed, 04 Dec 2019 00:00:00 +0000

At Vectops we use a combination of different technologies for different purposes. Ranging from homelabs, to full-blown multinational production systems that our staff works on during office hours (sometimes beyond office hours). We’ve tested and use daily different ways to connect to private infrastructures, ussually through a VPN connection.

This is where ZeroTier comes in, being a SD-WAN (Software Defined Wide Area Network) it doesn’t depend on new hardware, it can just be set up using whatever you already have, as long as you have more than one machine. Take into account that your phone counts :). This is basically a P2P network that only your devices can access.

ZeroTier’s virtual networks are defined by it’s creators as:

ZeroTier virtual networks are like chat rooms for machines. Just create virtual networks, join them from your devices and systems, approve authorization for the things you’ve added, and you’re done! Advanced features like our network rules engine help you manage your networks like a pro, and automatic end-to-end encryption keeps everything private and secure.

Registering

To be able to use ZeroTier you need to register on their website, this is so you can control the P2P network: ZeroTier. Click on Login and then Register:

After you register and confirm your email, you could purchase a subscription since the service doesn’t allow more than 100 devices on the free plan, also there’s no support if you’re not paying. That’s just how the world works, even though the service is free, people need to be paid for their work.

Let’s get started.

Creating Networks

You need to create a network for your machines (and phone) to connect, click on Networks and then on Create a Network. Take into account that even using the free plan you still get to be able to create unlimited networks here.

The network creation is immediate, it comes with a Network ID and a random name:

This network can be Private or Public.

Private networks are the safest way to go around, but also mean you have to make a couple of clicks per machine you add to the network.

Public networks allow anyone with the network ID to connect to it and be authorized.

Afterwards, you can set up the actual network.

Setting up the network

IPv4 ranges can be selected from the network control panel, it comes with the Auto-Assign from range option pre-selected and a random range selected too. Note that all of these ranges are private IPs, you can’t see them outside of your ZeroTier network. Since you’re using the free plan there’s no actual reason to use anything more than a /24 range, select one, in this case: 192.168.195.x:

These ranges also come with an established route, if you would like to change them you could do so as well.

In the case of IPv6 you can also Auto-Assign the IPs, depending on what you need you could use:

ZeroTier RFC4193 (/128 for each device)
ZeroTier 6PLANE (/80 routable for each device)
Auto-Assign from Range

Now to the machines:

Setting up our peers

We’re going to use a couple of examples here so you can see how to set up different machines on different OS’s, although ZeroTier’s documentation is pretty complete about it:

Linux machine

Make sure you have curl and gpg installed. Most modern distros come with gpg, but not always curl. Depending on your linux distro just use your package manager to install these. Once installed run this command:

curl -s 'https://raw.githubusercontent.com/zerotier/ZeroTierOne/master/doc/contact%40zerotier.com.gpg' | gpg --import && if z=$(curl -s 'https://install.zerotier.com/' | gpg); then echo "$z" | sudo bash; fi

There’s another method to install the client, daemon services, and repository but it doesn’t check for cryptographic signatures and relies fully on SSL, just a simple script:

curl -s https://install.zerotier.com | sudo bash

After the script is done just join the node with the zerotier-cli using your network ID:

zerotier-cli join d3ecf5726d62e220

And finally, authorize it from your control panel by checking on the checkbox that appears on your panel with the node ID (This node ID is shown on the terminal when the installation process finishes)

Android phone

Yes, we’re including this because not everyone has more than one machine and not everyone wants to create VMs and some people need a connection straight to their laptops or desktop machines from their phones:

Go on the Play Store and search for ZeroTier, install it and join a network, pretty easy, pretty straightforward.

Oh and don’t forget to authorize the phone.

Final set up

There’s no final set up, just go ahead and connect to your devices, doesn’t matter if you’re on the same Wifi or on a different country, as long as the machine as an internet connection you should be able to get to it.

Don’t forget to fill the names and/or description fields on the control panel for the machines, else you’re gonna have a bad time after it gets to more than a couple machines.

Good Luck!

Set up your machine to use kubectl

Wed, 04 Dec 2019 00:00:00 +0000

At Vectops we like to dwell on new techologies, which includes Kubernetes (K8s from now on). Most of the infrastructure we use on a daily basis is currently tied to K8s in one way or another. As a result this means that our machines have to be set up for K8s in one way or another to be able to perform different tasks on it. If you decide to run, test or become proficient on K8s you need to have kubectl set up on your environment.

K8s allows you to have different permissions and roles depending on your role regarding the cluster, whichever it may be: admin, mantainer, operator, developer, tester, etc.

This article won’t go into detail on RBAC (Role-Based Access Control), if you want to learn more about it go here.

For whatever task you need to perform on the cluster you do need some sort of way to interact with the cluster.

Enter kubectl

kubectl is a CLI tool that allows you to run commands that interact with K8s clusters. Wether it may be getting information from a specific pod, deployment, service, etc. to editing a definition for an instance.

It can be installed in different ways, lets focus on the ones that can be maintained easily, for instance from the OS’s package manager.

After you install kubectl you need to configure it. Just scroll down, you’ll get there. Above all, take into account the freedom of choice.

Linux

The easiest to install, just copy and paste these commands on the terminal for each of the distros.

Please take into account that you should check the contents of the repo file as well as the URLs, just to be safe

You’re going to use package management, snaps and straight binary download for the sake of completeness (JUST USE ONE OF THESE)

Debian/Ubuntu/other derivatives that use .deb

sudo apt-get update && sudo apt-get install -y apt-transport-https
curl -s https://packages.cloud.google.com/apt/doc/apt-key.gpg | sudo apt-key add -
echo "deb https://apt.kubernetes.io/ kubernetes-xenial main" | sudo tee -a /etc/apt/sources.list.d/kubernetes.list
sudo apt-get update
sudo apt-get install -y kubectl

CentOS/RHEL/Fedora/other derivatives that use .rpm

cat <<EOF > /etc/yum.repos.d/kubernetes.repo
[kubernetes]
name=Kubernetes
baseurl=https://packages.cloud.google.com/yum/repos/kubernetes-el7-x86_64
enabled=1
gpgcheck=1
repo_gpgcheck=1
gpgkey=https://packages.cloud.google.com/yum/doc/yum-key.gpg https://packages.cloud.google.com/yum/doc/rpm-package-key.gpg
EOF
yum install -y kubectl

Snap

sudo snap install kubectl --classic

Straight binary

Download the latest release from google:

curl -LO https://storage.googleapis.com/kubernetes-release/release/`curl -s https://storage.googleapis.com/kubernetes-release/release/stable.txt`/bin/linux/amd64/kubectl

Give it executable permissions

chmod +x ./kubectl

And move it to your binary path

sudo mv ./kubectl /usr/local/bin/kubectl

Make sure it works!

kubectl version

Windows

In the case of Windows you can decide to use a few tools including but not limited to WSL (Windows subsystem for linux), PowerShell, Chocolatey, etc.

WSL

Just use the same procedures for linux on your WSL console

PowerShell

PSGallery

You can use PSGallery:

Install-Script -Name install-kubectl -Scope CurrentUser -Force
install-kubectl.ps1 [-DownloadLocation $PATH]

You should change: $PATH for whichever path you want to choose for the download

Curl

Pure PowerShell allows you to run kubectl as well:

curl -LO https://storage.googleapis.com/kubernetes-release/release/v1.17.0/bin/windows/amd64/kubectl.exe

Don’t forget to add the .exe to your path, else you’ll have to navigate to the download path all the time or use an absolute route for the binary.

Chocolatey

choco install kubernetes-cli

Mac

The number of systems and devops engineers that use Mac either grows or diminishes every year (depends on who you ask, to each their own. I don’t judge). However there’s a lot our there that use it daily. In the same way that Linux and Windows users have a few ways to install it.

Homebrew

For Homebrew it’s a simple one-liner:

brew install kubectl

Macports

Same with Macports

sudo port selfupdate; sudo port install kubectl

Straight binary on MAC

The binary method for MAC is basically the same as for linux but with a different binary package:

curl -LO "https://storage.googleapis.com/kubernetes-release/release/$(curl -s https://storage.googleapis.com/kubernetes-release/release/stable.txt)/bin/darwin/amd64/kubectl"

Give it executable permissions

chmod +x ./kubectl

And move it to your binary path

sudo mv ./kubectl /usr/local/bin/kubectl

Make sure it works!

kubectl version

Configuration

After the installation is done you need to add the config file for kubernetes on your $PATH for kubectl to be able to work against your cluster.

On Linux, MAC and WSL this can be done by copying your config file on the following route:

~/.kube/config

In the case of Windows the file should be copied on the following route:

%USERPROFILE%/.kube/config

Checking that everything works

You can test that kubectl has access to the cluster by issuing the command:

kubectl version

If the connection works you’ll see an output like this:

Client Version: version.Info{Major:"1", Minor:"17", GitVersion:"v1.17.0", GitCommit:"70132b0f130acc0bed193d9ba59dd186f0e634cf", GitTreeState:"clean", BuildDate:"2019-12-10T03:03:57Z", GoVersion:"go1.13.5", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"16", GitVersion:"v1.16.3", GitCommit:"b3cbbae08ec52a7fc73d334838e18d17e8512749", GitTreeState:"clean", BuildDate:"2019-11-13T11:13:49Z", GoVersion:"go1.12.12", Compiler:"gc", Platform:"linux/amd64"}

Where the Client Version is the kubectl binary version and the Server Version is the Cluster version.

Final words

This tutorial should help you configure kubectl for your machine, it can also be used to set up a control server (just a simple virtual machine that has access to the same network as the K8s cluster)

In case you need any help you can type in a comment, we try to answer them ASAP.

Good Luck!

Updating a Proxmox Cluster from 4.4 to 5.4 the safe way

Wed, 04 Dec 2019 00:00:00 +0000

Most of us have had instances where a production virtualization cluster can’t be updated for whatever reasons: stability, fear of screwing up, orders from above, or scared co-workers that might think that once it’s in production you shouldn’t (just don’t) touch anything that might hinder the intended behaviour of a system.

However, there are some things that do need to be upgraded, especially in cases where a bug that has been pestering you has been fixed on a newer version or a new functionality has been added. Now, I’m not one of those guys that likes to have absolute bleeding-edge software on production servers (obvious reasons might include: it just hasn’t been tested long enough), but having software on production that might soon reach End Of Life or that isn’t behaving up to current standards, that, I have a problem with.

I started to work on a new company not long ago and I came to work on the first day to see a few Proxmox clusters running on 4.4, which isn’t that bad (don’t touch production servers remember?), so I tested the functionality and I came to see bugs I hadn’t seen in years (including issues with the no-vnc conections to the VMs) and decided to do something about it. So I convinced my manager to upgrade the clusters.

At the time of writing this article the current version of Proxmox is 6.0. I’ve had bad experiences upgrading to the first major version of Proxmox before so I decided to upgrade to 5.4 atleast until a more polished version of Proxmox comes arround.

The procedure is pretty simple, if you have classical shared storage (NFS, iSCSI disks, etc.).

Beware CEPH Users

This article doesn’t cover the steps for a CEPH upgrade, so please be careful

So, we start with a cluster. We need to balance our the whole cluster so we can get an empty node. That means migrating all the VMs to other nodes, all while making sure the target nodes don’t run out of RAM and in my case making sure the CPU load wasn’t topping out (production servers remember?).

The migration procedure is pretty straightforward:

Right click on a VM -> Migrate -> Select the target node -> Get a cup of coffee

After you finish this process you can go ahead and upgrade the node.

First off, start with a check from the terminal so we can make sure there’s no VMs running on the node:

qm list

Once we make sure there are no running VMs and we’re sure the node is empty (go ahead, check again.) we can proceed.

First step is to do an update on the repos with:

apt update

Then we can upgrade and reboot the current system with the newest kernel for 4.4:

apt upgrade -y; reboot

Once the node comes back up we need to add-in the new repos for the upgrade, as well as the GPG signatures for the repos:

sed -i 's/jessie/stretch/g' /etc/apt/sources.list
sed -i 's/jessie/stretch/g' /etc/apt/sources.list.d/pve-enterprise.list
wget http://download.proxmox.com/debian/proxmox-ve-release-5.x.gpg -O /etc/apt/trusted.gpg.d/proxmox-ve-release-5.x.gpg

Then we proceed with updating the repos on APT:

apt update

Now, this is very important, we need to remote sysv-rc, openrc, etc from the system so we’re able to get the latest systemd graceness on the system DO NOT REBOOT THE SYSTEM:

apt purge insserv sysv-rc initscripts openrc

DO NOT REBOOT THE SYSTEM Seriously, unless you wanna spend hours fixing the system or just formatting it and adding it as a new node. (which in hindsight isn’t such a bad idea)

After the removal is done we can finally upgrade the node to the 5.4 version of Proxmox:

apt-get dist-upgrade

This step can take a while, and it’s going to prompt you for answers a couple of times regarding configuration files, you can leave them as-is or you can upgrade them, up to you. I leave them as-is in case I need to recover anything.

Finally, you can reboot the system:

reboot

Et voilà!

Now you can migrate the VMs back to the node and start all over again with another node. This process can take a while and you might be able to automate it, however I’ve heard some stories both from RL and online from people that have had some bad luck with automating it and end up screwing up the cluster in some way or another.

Good Luck!

About us

Mon, 01 Jan 0001 00:00:00 +0000

We are just three Sysadmins that have had to work on different areas of IT, whether it would be infrastructure, networking, development, testing, etc. You know, pure Sysadmin stuff. After a few years of this, we have noticed that on-point documentation and easy to read use cases are more often than not just not there.

Thus, we have decided to start posting some of those use cases and articles, digested and easy to read so you don’t have to spend countless hours googling for stuff.

As we have.

Search

Mon, 01 Jan 0001 00:00:00 +0000