Simple Log Service:Install, run, upgrade, and uninstall Logtail

Install Logtail

You can install Logtail using a one-click method or a manual method. Simple Log Service supports one-click installation of Logtail only on ECS instances that are in the same region and belong to the same Alibaba Cloud account as the Simple Log Service project. In other cases, you must install Logtail manually.

# Download over the internal network
wget http://logtail-release-${region_id}.oss-${region_id}-internal.aliyuncs.com/linux64/logtail.sh -O logtail.sh; 
# Download over the internet
wget http://logtail-release-${region_id}.oss-${region_id}.aliyuncs.com/linux64/logtail.sh -O logtail.sh; 

# Install over the internet
chmod +x logtail.sh; ./logtail.sh install ${region_id}-internet
# If your servers are in self-managed data centers or from cloud providers outside China, data transfer over the internet may cause high network latency and instability. We recommend using transfer acceleration for data transfer.
chmod +x logtail.sh; ./logtail.sh install ${region_id}-acceleration
# Install over the internal network. This applies if your self-managed IDC has internal network access.
chmod +x logtail.sh; ./logtail.sh install ${region_id}

# Install over the internet
chmod +x logtail.sh; ./logtail.sh install-local ${region_id}-internet
# If your servers are in self-managed data centers or from cloud providers outside China, data transfer over the internet may cause high network latency and instability. We recommend using transfer acceleration for data transfer.
chmod +x logtail.sh; ./logtail.sh install-local ${region_id}-acceleration

# Install over the internet
.\logtail_installer.exe install ${region_id}-internet
# If your servers are in self-managed data centers or from cloud providers outside China, data transfer over the internet may cause high network latency and instability. We recommend using transfer acceleration for data transfer.
.\logtail_installer.exe install ${region_id}-acceleration

Batch installation of Logtail

You can install Logtail in batches in one of the following two ways:

• OOS orchestration: Suitable for scenarios with permission requirements. It supports high concurrency and is ideal for large-scale batch operations. For more information, see Use OOS to install or upgrade Logtail in batches.

• ECS Cloud Assistant: Easy to use. It lets you run temporary tasks directly using commands. The procedure is as follows:

1. ECS console - Storage Capacity Units

2. In the top navigation bar, select the region and resource group of the resource that you want to manage.

3. In the upper-right corner of the ECS Cloud Assistant page, click Create/Run Command.

4. In the Create Command panel, enter the installation command in the Command Content field. This example uses the Internet installation method. For more installation commands, see Install Logtail.

   

   The installation command used here is as follows:

   #!/bin/bash
   region_id='cn-hangzhou'
   wget http://logtail-release-${region_id}.oss-${region_id}.aliyuncs.com/linux64/logtail.sh -O logtail.sh
   chmod +x logtail.sh
   ./logtail.sh install ${region_id}-internet

   Important

   After you install Logtail, if you switch the network type of the ECS instance from classic network to VPC, you must update the Logtail configuration. For more information, see How do I update the machine group configuration after switching an ECS instance from the classic network to a VPC?.

5. In the Select Instances section, confirm that the Agent status of the target instance is Normal. If the status is not Normal, see Install Cloud Assistant Agent to install the Cloud Assistant Agent. Select the target instance and click Run. The execution status changes to Execution Succeeded. If the execution fails, see View execution results and fix common issues.

Start and stop Logtail

View the Logtail status and version

cat /usr/local/ilogtail/app_info.json

{
   "logtail_version" : "0.16.30",
}

{
    "logtail_version" : "1.0.0.0"
}

Upgrade Logtail

stop successfully
Stop logtail successfully.
Upgrading logtail files ...
Upgrade logtail files successfully.
Starting logtail ...
ilogtail is running
Upgrade logtail successfully.
{
        "UUID" : "XXXXXXXX-XXXX",
        "compiler" : "GCC 9.3.1",
        "hostname" : "xxx",
        "instance_id" : "XXXXXXXX-XXXX_172.16.0.75_1730950372",
        "ip" : "172.16.0.75",
        "logtail_version" : "2.0.8",
        "os" : "Linux; 5.10.134-13.an8.x86_64; #1 SMP Mon Jan 9 10:39:46 CST 2023; x86_64",
        "update_time" : "2024-11-07 11:32:52"
}

Uninstall Logtail

wget http://logtail-release-${region_id}.oss-${region_id}.aliyuncs.com/linux64/logtail.sh -O logtail.sh; chmod +x logtail.sh; ./logtail.sh uninstall

.\logtail_installer.exe uninstall

Install the Logtail component

If you use an ACK cluster that belongs to the same Alibaba Cloud account as your Simple Log Service project, follow the instructions for ACK clusters. Otherwise, follow the instructions for self-managed clusters.

View the Logtail status, version, and IP address

• Run the following command to view the Logtail status.

  kubectl get po -n kube-system | grep logtail

  The following result is returned:

  NAME            READY     STATUS    RESTARTS   AGE
  logtail-ds-gb92k   1/1       Running   0          2h
  logtail-ds-wm7lw   1/1       Running   0          4d

• Run the following command to view the Logtail version number, IP address, and other information.

  kubectl exec logtail-ds-gb92k -n kube-system cat /usr/local/ilogtail/app_info.json

  The following result is returned:

  {
     "hostname" : "logtail-ds-gb92k",
     "instance_id" : "0EBB2B0E-0A3B-11E8-B0CE-0A58AC140402_172.20.4.2_1517810940",
     "ip" : "192.0.2.0",
     "logtail_version" : "0.16.2",
     "os" : "Linux; 3.10.0-693.2.2.el7.x86_64; #1 SMP Tue Sep 12 22:26:13 UTC 2017; x86_64",
     "update_time" : "2021-02-05 06:09:01"
  }

Upgrade and roll back Logtail

1. Before upgrading, back up the description files related to the Logtail component.

   Important

   During the upgrade, which takes a few seconds, a small amount of data may be collected repeatedly or lost because the Logtail container restarts.

   kubectl get ds -n kube-system logtail-ds -o yaml > logtail-ds.yaml
   kubectl get deployment -n kube-system alibaba-log-controller -o yaml > alibaba-log-controller.yaml
   kubectl get crd aliyunlogconfigs.log.alibabacloud.com -o yaml > aliyunlogconfigs-crd.yaml
   kubectl get cm -n kube-system alibaba-log-configuration -o yaml > alibaba-log-configuration.yaml
   kubectl get aliyunlogconfigs --all-namespaces -o yaml > aliyunlogconfigs-cr.yaml

2. Choose the component upgrade method based on your cluster type. If you use an ACK cluster that belongs to the same Alibaba Cloud account as your Simple Log Service project, follow the instructions for ACK clusters. Otherwise, follow the instructions for self-managed clusters.

   Upgrade method for ACK clusters

   In most cases, we recommend using the automatic upgrade method. If you have modified parameters (such as environment variables) in the logtail-ds DaemonSet or the alibaba-log-controller deployment, use the manual upgrade method to prevent your changes from being reset.

   Automatic upgrade

   Important

   Automatic upgrades will reset any manual configurations you have made in logtail-ds and alibaba-log-controller.

   1. Log on to the ACK console.

   2. On the Clusters page, find the target cluster and choose More > Operations > Manage Components in the Actions column.

   3. On the Logs And Monitoring tab, find logtail-ds and click Upgrade.

   4. In the Upgrade Component dialog box, click OK.

      Important

      If you cannot upgrade to the latest version of Logtail, it may be because your Kubernetes cluster version is too old. Upgrade your Kubernetes cluster first or use the manual upgrade method.

      After performing the upgrade, you can view the logtail-ds pod status in the ACK console. If all logtail-ds pods are in the Running state, the upgrade is successful.

   Manual upgrade

   Important

   Manual upgrades do not update your configuration based on the latest Logtail component. Some feature optimizations may not be available.

   Manual upgrades include upgrading logtail-ds and alibaba-log-controller. In most cases, you only need to upgrade logtail-ds to obtain the collection capabilities provided by the new version of Logtail. When you need to obtain the collection capabilities of the new Logtail CRD method, you must upgrade alibaba-log-controller. The following steps use logtail-ds as an example.

   1. Log on to the ACK console.

   2. On the Clusters page, find the target cluster and choose More > Operations > Manage Components in the Actions column.

   3. Choose Workloads > DaemonSets.

      Note

      When you want to upgrade alibaba-log-controller, choose Workloads > Deployments. Then, in the kube-system namespace, find alibaba-log-controller and complete the upgrade.

   4. Set Namespace to kube-system, and then click Edit for logtail-ds.

   5. Check whether the following environment variables exist.

      If the ALIYUN_LOGTAIL_CONFIG, ALIYUN_LOGTAIL_USER_ID, and ALIYUN_LOGTAIL_USER_DEFINED_ID environment variables do not exist, it may be because your Logtail version is too old. You can submit a ticket to inquire about the upgrade method.

   6. Click Select Image Tag for Image Tag.

   7. In the Image Tag dialog box, click the latest version, and then click OK.

   8. On the right side of the page, click Update.

      After performing the upgrade, you can view the logtail-ds pod status in the ACK console. If all logtail-ds pods are in the Running state, the upgrade is successful.

   Upgrade method for self-managed clusters

   Note

   We recommend that you update by installing the latest Logtail component. Updating only the image version number of some components (such as logtail-ds, alibaba-log-controller) may cause the upgrade to fail.

   Reinstall the Logtail component to complete the automatic upgrade. For more information, see Install the Logtail component.

3. If you want to roll back to a specific version, follow these steps.

   Note

   The YAML files backed up before the upgrade contain a lot of redundant information that you must manually delete before you can use them to restore the Logtail configuration. You can use the kubectl-neat tool to do this. The fields to be deleted are metadata.creationTimestamp, metadata.generation, metadata.resourceVersion, metadata.uid, and status.

   1. Determine whether to keep the new Logtail configuration after the upgrade based on your business needs.

      If you do not need to keep it, you can delete the new Logtail configuration after the upgrade.

   2. Delete the redundant information from the backup files.

      cat logtail-ds.yaml | kubectl-neat > neat-logtail-ds.yaml
      cat alibaba-log-controller.yaml | kubectl-neat > neat-alibaba-log-controller.yaml
      cat aliyunlogconfigs-crd.yaml | kubectl-neat > neat-aliyunlogconfigs-crd.yaml
      cat alibaba-log-configuration.yaml | kubectl-neat > neat-alibaba-log-configuration.yaml
      cat aliyunlogconfigs-cr.yaml | kubectl-neat > neat-aliyunlogconfigs-cr.yaml

   3. Apply the streamlined backup files to restore the Logtail configuration.

      kubectl apply -f neat-logtail-ds.yaml
      kubectl apply -f neat-alibaba-log-controller.yaml
      kubectl apply -f neat-aliyunlogconfigs-crd.yaml
      kubectl apply -f neat-alibaba-log-configuration.yaml
      kubectl apply -f neat-aliyunlogconfigs-cr.yaml

Uninstall Logtail

Choose the component uninstallation method based on your cluster type. If you use an ACK cluster that belongs to the same Alibaba Cloud account as your Simple Log Service project, follow the uninstallation method for ACK clusters. If you use a self-managed cluster, or if your ACK cluster and Simple Log Service project belong to different Alibaba Cloud accounts, follow the uninstallation method for self-managed clusters.

How to view the image version?

You can view the image version in the image repository.

• logtail-ds image repository

• alibaba-log-controller image repository

How to use one Simple Log Service project for multiple Kubernetes clusters?

If you want to collect container logs from multiple clusters into the same Simple Log Service project, keep the installation parameters consistent with those used for the first cluster's component installation when you install the components on other clusters.

How to view Logtail logs?

Logtail logs are stored in the /usr/local/ilogtail/ directory in the Logtail container. The file names are ilogtail.LOG and logtail_plugin.LOG.

The standard output in the Logtail container is not a valid reference. You can ignore the following standard output content.

start umount useless mount points, /shm$|/merged$|/mqueue$
umount: /logtail_host/var/lib/docker/overlay2/3fd0043af174cb0273c3c7869500fbe2bdb95d13b1e110172ef57fe840c82155/merged: must be superuser to unmount
umount: /logtail_host/var/lib/docker/overlay2/d5b10aa19399992755de1f85d25009528daa749c1bf8c16edff44beab6e69718/merged: must be superuser to unmount
umount: /logtail_host/var/lib/docker/overlay2/5c3125daddacedec29df72ad0c52fac800cd56c6e880dc4e8a640b1e16c22dbe/merged: must be superuser to unmount
......
xargs: umount: exited with status 255; aborting
umount done
start logtail
ilogtail is running
logtail status:
ilogtail is running

How to check the status of Simple Log Service related components in a Kubernetes cluster?

Run the following commands to check the status.

kubectl get deploy alibaba-log-controller -n kube-system
kubectl get ds logtail-ds -n kube-system

What should I do if alibaba-log-controller fails to start?

Confirm that you have followed these installation instructions.

• Run the installation command on the master node of the Kubernetes cluster.

• Enter your cluster ID in the installation command parameters.

If the installation fails because of these issues, use the kubectl delete -f deploy command to delete the generated installation template and run the installation command again.

How to check the status of the Logtail-ds DaemonSet in a Kubernetes cluster?

Run the kubectl get ds -n kube-system command to view the Logtail-ds DaemonSet status.

How to view the running logs of Logtail?

Logtail running logs are saved in the /usr/local/ilogtail/ directory. The file name is ilogtail.LOG. Rotated files are compressed and stored as ilogtail.LOG.x.gz. For example, run the following command to view the logs.

kubectl exec logtail-ds-gb92k -n kube-system tail /usr/local/ilogtail/ilogtail.LOG

The following result is returned:

[2018-02-05 06:09:02.168693] [INFO] [9] [build/release64/sls/ilogtail/LogtailPlugin.cpp:104] logtail plugin Resume:start
[2018-02-05 06:09:02.168807] [INFO] [9] [build/release64/sls/ilogtail/LogtailPlugin.cpp:106] logtail plugin Resume:success
[2018-02-05 06:09:02.168822] [INFO] [9] [build/release64/sls/ilogtail/EventDispatcher.cpp:369] start add existed check point events, size:0
[2018-02-05 06:09:02.168827] [INFO] [9] [build/release64/sls/ilogtail/EventDispatcher.cpp:511] add existed check point events, size:0 cache size:0 event size:0 success count:0

How to restart Logtail in a specific pod?

1. Stop Logtail.

   logtail-ds-gb92k is the container name and kube-system is the namespace. Replace them as needed.

   kubectl exec logtail-ds-gb92k -n kube-system /etc/init.d/ilogtaild stop

2. The following output indicates that Logtail is stopped successfully.

   kill process Name: ilogtail pid: 7
   kill process Name: ilogtail pid: 9
   stop success

3. Start Logtail.

   logtail-ds-gb92k is the container name and kube-system is the namespace. Replace them as needed.

   kubectl exec logtail-ds-gb92k -n kube-system /etc/init.d/ilogtaild start

The following output indicates that Logtail is started successfully.

ilogtail is running