Chapter 4
Implementation of Monitoring and Auto-Ballooning

For the purpose of this thesis, we describe the implementation and experimentally evaluate only the monitoring and autoballooning parts of the DRS. The following sections describe the implementation details of these two components.

4.1 Monitoring

As we discussed in Chapter 3, the monitoring component runs on each host and monitors the host and the guests running on that host to detect hotspots. The next few sections discuss which metrics are monitored, why they are monitored, and how they are monitored. Our implementation is done in the Python programming language. The monitoring service runs periodically in every fixed time interval which is configurable. The default interval is set to then seconds. We will refer to this time interval as the monitor interval in the rest of this document.

4.1.1 Host Monitoring

Monitoring the host is simple because the Linux kernel exposes much of the needed information stored in its internal data structure through procfs [22]. Procfs is a filesystem in the Linux kernel, and hence can be accessed using the standard filesystem syscalls or through higher level API’s that exist in almost all programming languages for this purpose. Since the DRS is running on the host, it has unhindered access to all the information in procfs.

CPU Monitoring

For monitoring the CPU usage, we use the information in the /proc/stat file. The /proc/stat file has the following fields capturing how much time CPU spent doing what task [22].

The time is measured in kernel jiffies. Jiffy is a counter that ticks on every timer interrupt. The interrupt frequency is 100Hz in majority of the systems. From the above values, we can get the total time that has passed, and time for which the cpu was busy.

TotalTime = user + nice + system + idle + iowait + irq + iowait + irq + softirq + steal + guest + guest_nice

BusyTime = user + nice + system + irq + softirq + guest + guest_nice

The TotalTime calculated here is the total time since the start of the system. Hence this will give the average cpu usage since the start of the system, which is not a useful metric for us. The average cpu usage in the previous monitor interval is more useful. To calculate that, we keep the TotalTime and BusyTime at the previous monitor interval and subtract those values from the current values. So,

PrecentageCpuUsage = BusyTime PreviousBusyTime TotalTime PreviousTotalTime 100

Memory Monitoring

Memory usage information is present in the /proc/meminfo file [22]. For monitoring, we need several memory metrics which are described below.

4.1.2 Guest Monitoring

Monitoring a VM is more complex than monitoring the host because some of the metrics needed are not exposed by the VM to the host in any conventional way. Host operating system just gets to see the process abstraction of each virtual machine. So, the host has only as much information about each guest as it has about the rest of the processes running on it. In the next two sections, along with which metrics have been monitored, we have discuss some new techniques which we use to get some of the information from the guest VMs.

CPU Monitoring

CPU monitoring for a guest VM is straightforward because the host OS has the scheduling information for each process it is running. This information is exposed by the procfs. The metrics related to CPU utilization measured are discussed below.

Memory Monitoring

Getting the memory usage statistics is difficult because the host has no information about the memory management of the guest. The host only knows the amount of memory allocated by each guest. For calculating the idle memory of each guest, we need the amount of available memory inside the VM.

One way for a guest VM to expose some information to the host is through a device driver. In a virtualized environment, all the devices available to a VM are virtual devices which are emulated by the hypervisor. So, a device driver running inside the VM can send some information to the device which is emulated by the hypervisor. The virtio balloon driver in the Linux kernel has a way of exposing the memory statistics to the host. It gives the total memory, free memory, swap in, major page faults, minor page fault metrics. We have modified the balloon driver to expose the available memory metric to the host too. We have also modified the backend virtio balloon hardware in QEMU to take the Available Memory metric from the balloon driver inside the guest. The following metrics related to the memory usage of the VMs are monitored. Figure 4.1 shows the relative sizes of different memory metrics calculated.


PIC

Figure 4.1: Relative sizes of different types of memory metrics

4.1.3 Hotspot Detection and Key-Value Store Updation

The hotspot detection algorithm should be robust enough not to generate false alarms. A simple threshold based algorithm which takes absolute or average values into account can raise many false alarms. Andreolini et al. [24] have shown the drawbacks of such algorithms and proposed a more robust statistical model to detect changes in the load profile of a machine which is based on the CUSUM (Cumulative Sum) algorithm [25]. We have used this algorithm for detecting hotspots and the time to update the key-value store. The algorithm is described below for the sake of completeness.

The hotspots due to memory overload are detected using the load memory metric of the host. The values for load memory are recorded in ever monitor interval to form a time series. Exponential average of the data is calculated as

μi = α loadmemi + (1 α)μi1

The value of α has been chosen to be 0.1. The algorithm detects abrupt increase in μi using di.

d0 = 0; di = di1 + (loadmemi (μi + K))

di measures all the deviations from μi that are greater than K. K = Δ 2 where Δ is the minimum shift to be detected. It has been set to (0.005 TotalMemory) to detect minimum 1% change in the value of loadmem. Change in the load profile of the host is triggered when di > H where H = hσ, h being a design parameter and σ being the standard deviation of the time series upto that point. We have set h = 7, which means that the load profile change is detected in an average of 14 samples [24]. When the load-profile changes, the key-value store is updated with totalmem loadmem. When the load profile changes and (μi > 0.8 totalmem), hotspot is triggered and the migration service becomes active.

The hotspots in the CPU usage are also calculated in a similar way. The only difference is that the CUSUM algorithm runs on the steal time of all the guests, and any guest can trigger hotspot when its load profile changes and its AverageStealTime > 10%. Additionally, the CUSUM algorithm runs on the busy time of the host to trigger the updation of the key-value store. The value stored in the key-value store is (NumberOfCPUs 100 BusyTimePercentage)

4.2 Auto-Ballooning

Ballooning is a technique for increasing or decreasing the current memory of a guest. Auto-Ballooning is the process of automatically balancing memory amongst multiple guests running on a system by taking some memory from the idle guests and giving it to the needy guests. Auto-Ballooning is the most essential component for memory overcommitment.

4.2.1 Hard Ballooning and Soft Ballooning

Depending upon the type of memory reclaimed ballooning can be classified into two types - hard ballooning and soft ballooning. Soft ballooning is the process of reclaiming memory which is free inside the guest while hard ballooning is the process of reclaiming memory which is used inside the guest.

SoftLowerBound = max(UsedMemory,GuestReserved) SoftIdleMemory = max(AllocatedMemory SoftLowerBound, 0) HardLowerBound = max(LoadMemory,GuestReserved) HardIdleMemory = max(UsedMemory HardLowerBound, 0)

For soft ballooning, the guest is ballooned down to (CurrentMemory SoftIdleMemory), which will reclaim the SoftIdleMemory. To reclaim HardIdleMemory, the guest has to be ballooned down to (UsedMemory HardIdleMemory). Thus, ballooning down a guest from current memory to (CurrentMemory SoftIdleMemory) will reclaim SoftIdleMemory. After this, ballooning down from (CurrentMemory SoftIdleMemory) to UsedMemory will reclaim no memory. Then, ballooning down from used memory to (UsedMemory HardIdleMemory) will reclaim the HardIdleMemory. Figure 4.2 shows the two different types of ballooning.


PIC

Figure 4.2: Soft and hard ballooning

4.2.2 Auto-Ballooning Algorithm

The autoballooning algorithm first identifies the guests with idle memory and the guests which need more memory. We have already described the method of calculating the idle memory earlier. We identify needy guests as the ones whose average load memory is greater than a certain threshold. We have kept the threshold to 90% of the current memory of the guest. Needy guests are ballooned up in intervals of 10% of their current memory. Total needed memory is the sum total of the memory needed by all the needy guests(which is 10% of the current memory for each needy guest).

isNeedy(guest) = TrueLoadMemory 0.9 CurrentMemoryF alse otherwise

NeededMemory = 0.1 CurrentMemory

The host unused memory is the amount of memory on the host which is neither used by/allocated to any guest nor is being used by the host, and hence, can be given away to any host by ballooning it up. Ballooning up a guest reduces the host unused memory, while ballooning a guest down decreases it. It is calculated as follows.

HostUnusedMemory = AvailableMemory HypervisorExtra or HostUnusedMemory = TotalMemory LoadMemory TotalIdleMemory

Auto-Ballooning can be triggered in two cases - (HostUnusedMemory < 0.1 TotalMemory) or (TotalNeededMemory > 0). The first case is important to keep the amount of swapped memory on the host low. To handle the first case, first the soft idle memory and then the hard idle memory form the guests is ballooned out till the (HostUnusedMemory < 0.2 TotalMemory). If the idle memory is exhausted before this 20% memory becomes unused, swapping is inevitable and the job of resolving it is left to the migration service.

In the second case, either there is a needy guest or memory needs to be reclaimed for a guest which would be migrated to the machine. Both the situations are similar except that when memory is reclaimed for an incoming guest, there is no needy guest to be ballooned up. Depending on the needed memory and idle memory, three situations can arise.

1.
TotalNeededMemory ≤ TotalSoftIdleMemory. The requirements of each needy guest can be satisfied by just soft-ballooning. So, the idle guests are ballooned down to reclaim the needed memory and then, the needy guests are ballooned up.
2.
TotalSoftIdleMemory < TotalNeededMemory ≤ TotalHardIdleMemory. First, all the soft idle memory is ballooned out. Then, the rest of the needed memory is divided among the guests with hard idle memory in proportion of their hard idle memory. So, memory reclaimed by hard ballooning for a guest is given by NeedAfterSoftBallooning = TotalNeededMemory − TotalSoftIdleMemory

HardReclaim = GuestHardIdleMem ∗ NeedAfterSoftBallooning TotalHardIdleMemory

Here, GuestHardIdleMem is the hard idle memory for that particular guest, while TotalHardIdleMem is the total hard idle memory. The reason hard ballooning is treated differently from soft ballooning is because soft ballooning is not supposed to have any affect on the performance of the guest as it takes away only the free pages. On the other hand, hard ballooning also reclaims some of the page caches, which might have some effect on the performance of the guest.

3.
TotalNeededMemory > TotalHardIdleMemory. This case implies that there is a hotspot on the host. Since the demands of all the guests cannot be satisfied, the memory is given to them or reclaimed from them based on their entitlement. The entitled memory of each guest is calculated as EntitledMemory = GuestMaximumMemory ∗ MemoryOvercommitmentRatio

After this, the idle memory and needed memory calculation is done again.

IdleMemory = max(CurrentMemory − EntitledMemory, 0)

NeededMemory = 0if !isNeedy(guest) 0CurrentMemory > EntitledMemory min(0.1 ∗ CurrentMemory, CurrentMemory − EntitledMemory)otherwise

Then, the idle memory is ballooned out of the guests and the needed memory is provided to the needy guests.

Summary

In this chapter, we looked at the different techniques we use to monitor various metrics for hosts and guests. These metrics are used to trigger hotspot and make the ballooning service active. We also looked at the CUSUM based algorithm used to filter out unecessary spikes in the resource usage profiles of the host. In the end, we also discussed our technique of autoballooning.

  • LinkedIn
  • Tumblr
  • Reddit
  • Google+
  • Pinterest
  • Pocket