How do I get help?#
Having trouble? We would like to help!
Where are the logs?#
Cloud-init uses two files to log to:
/var/log/cloud-init-output.log: captures the output from each stage of cloud-init when it runs
/var/log/cloud-init.log: very detailed log with debugging output, detailing each action taken
/run/cloud-init: contains logs about how cloud-init decided to enable or disable itself, as well as what platforms/datasources were detected. These logs are most useful when trying to determine what cloud-init ran or did not run.
Be aware that each time a system boots, new logs are appended to the files in /var/log. Therefore, the files may have more than one boot worth of information present.
When reviewing these logs look for any errors or Python tracebacks to check for any errors.
Where are the configuration files?#
Cloud-init config is provided in two places:
These files can define the modules that run during instance initialization, the datasources to evaluate on boot, and other settings.
Where are the data files?#
Inside the /var/lib/cloud/ directory there are two important subdirectories:
The /var/lib/cloud/instance directory is a symbolic link that points to the most recently used instance-id directory. This folder contains the information cloud-init received from datasources, including vendor and user data. This can be helpful to review to ensure the correct data was passed.
It also contains the datasource file that contains the full information about what datasource was identified and used to setup the system.
Finally, the boot-finished file is the last thing that cloud-init does.
The /var/lib/cloud/data directory contain information related to the previous boot:
instance-id: id of the instance as discovered by cloud-init. Changing this file has no effect.
result.json: json file will show both the datasource used to setup the instance, and if any errors occurred
status.json: json file shows the datasource used and a break down of all four modules if any errors occurred and the start and stop times.
What datasource am I using?#
To correctly setup an instance, cloud-init must correctly identify the cloud that it is on. Therefore knowing what datasource is used on an instance launch can help aid in debugging.
To find what datasource is getting used run the cloud-id command:
$ cloud-id nocloud
If the cloud-id is not what is expected, then running the ds-identify script in debug mode and providing that in a bug can help aid in resolving any issues:
$ sudo DEBUG_LEVEL=2 DI_LOG=stderr /usr/lib/cloud-init/ds-identify --force
The force parameter allows the command to be run again since the instance has already launched. The other options increase the verbosity of logging and put the logs to STDERR.
How can I re-run datasource detection and cloud-init?#
If a user is developing a new datasource or working on debugging an issue it may be useful to re-run datasource detection and the initial setup of cloud-init.
To do this, force ds-identify to re-run, clean up any logs, and re-run cloud-init:
$ sudo DI_LOG=stderr /usr/lib/cloud-init/ds-identify --force $ sudo cloud-init clean --logs $ sudo cloud-init init --local $ sudo cloud-init init
These commands will re-run cloud-init as if this were first boot of a system: this will, at the very least, cycle SSH host keys and may do substantially more. Do not run these commands on production systems.
How can I debug my user data?#
Two of the most common issues with user data, that also happens to be cloud-config is:
Incorrectly formatted YAML
First line does not contain #cloud-config
To verify your YAML, we do have a short script called validate-yaml.py that can validate your user data offline.
Another option is to run the following on an instance to debug userdata provided to the system:
$ cloud-init schema --system --annotate
As launching instances in the cloud can cost money and take a bit longer, sometimes it is easier to launch instances locally using Multipass or LXD:
Why did cloud-init never complete?#
To check if cloud-init is running still, run:
$ cloud-init status
To wait for cloud-init to complete, run:
$ cloud-init status --wait
There are a number of reasons that cloud-init might never complete. This list is not exhaustive, but attempts to enumerate potential causes:
failed dependent services in the boot
bugs in the kernel or drivers
bugs in external userspace tools that are called by cloud-init
a command in
runcmdthat never completes (ex: running cloud-init status –wait will wait forever on itself and never complete)
nonstandard configurations that disable timeouts or set extremely high values (“never” is used in a loose sense here)
Failing to Complete on Systemd:#
Cloud-init consists of multiple services on systemd. If a service that cloud-init depends on stalls, cloud-init will not continue. If reporting a bug related to cloud-init failing to complete on systemd, please make sure to include the following logs.
$ systemd-analyze critical-chain cloud-init.target $ journalctl --boot=-1 $ systemctl --failed
How can I make a module run on every boot?#
Modules have a default frequency that can be overridden. This is done
by modifying the module list in
Change the module from a string (default) to a list.
Set the first list item to the module name and the second item to the frequency.
The following example demonstrates how to log boot times to a file every boot.
cloud_final_modules: # list shortened for brevity - [phone-home, always] - final-message - power-state-change
Then your userdata could then be:
#cloud-config phone_home: url: http://example.com/$INSTANCE_ID/ post: all
How can I test cloud-init locally before deploying to the cloud?#
Several different virtual machine and containerization tools can be used for testing locally. Multipass, LXD, and qemu are described in this section.
Multipass is a cross-platform tool to launch Ubuntu VMs across Linux, Windows, and macOS.
When a user launches a Multipass VM, user data can be passed by adding the –cloud-init flag and the appropriate YAML file containing user data:
$ multipass launch bionic --name test-vm --cloud-init userdata.yaml
Multipass will validate the YAML syntax of the cloud-config file before attempting to start the VM! A nice addition to help save time when experimenting with launching instances with various cloud-configs.
Multipass only supports passing user-data and only as YAML cloud-config files. Passing a script, a MIME archive, or any of the other user-data formats cloud-init supports will result in an error from the YAML syntax validator.
LXD offers a streamlined user experience for using linux system containers. With LXD, a user can pass:
The following initializes a container with user data:
$ lxc init ubuntu-daily:bionic test-container $ lxc config set test-container user.user-data - < userdata.yaml $ lxc start test-container
To avoid the extra commands this can also be done at launch:
$ lxc launch ubuntu-daily:bionic test-container --config=user.user-data="$(cat userdata.yaml)"
Finally, a profile can be setup with the specific data if a user needs to launch this multiple times:
$ lxc profile create dev-user-data $ lxc profile set dev-user-data user.user-data - < cloud-init-config.yaml $ lxc launch ubuntu-daily:bionic test-container -p default -p dev-user-data
The above examples all show how to pass user data. To pass other types of configuration data use the config option specified below:
The cloud-localds command from the cloud-utils package generates a disk with user supplied data. The NoCloud datasouce allows users to provide their own user data, metadata, or network configuration directly to an instance without running a network service. This is helpful for launching local cloud images with QEMU for example.
The following is an example of creating the local disk using the cloud-localds command:
$ cat >user-data <<EOF #cloud-config password: password chpasswd: expire: False ssh_pwauth: True ssh_authorized_keys: - ssh-rsa AAAA...UlIsqdaO+w== EOF $ cloud-localds seed.img user-data
The resulting seed.img can then get passed along to a cloud image containing cloud-init. Below is an example of passing the seed.img with QEMU:
$ qemu-system-x86_64 -m 1024 -net nic -net user \ -hda ubuntu-20.04-server-cloudimg-amd64.img \ -hdb seed.img
The now booted image will allow for login using the password provided above.
For additional configuration, users can provide much more detailed configuration, including network configuration and metadata:
$ cloud-localds --network-config=network-config-v2.yaml \ seed.img userdata.yaml metadata.yaml
Where can I learn more?#
Below are some videos, blog posts, and white papers about cloud-init from a variety of sources.