Node e2e tests are component tests meant for testing the Kubelet code on a custom host environment.
Tests can be run either locally or against a host running on GCE.
Node e2e tests are run as both pre- and post- submit tests by the Kubernetes project.
Note: Linux only. Mac and Windows unsupported.
Note: There is no scheduler running. The e2e tests have to do manual scheduling, e.g. by using
Why run tests Locally? Much faster than running tests Remotely.
- Install etcd on your PATH
- Verify etcd is installed correctly by running
- Or make etcd binary available and executable at
- Install ginkgo on your PATH
- Verify ginkgo is installed correctly by running
From the Kubernetes base directory, run:
This will: run the ginkgo binary against the subdirectory test/e2e_node, which will in turn: - Ask for sudo access (needed for running some of the processes) - Build the Kubernetes source code - Pre-pull docker images used by the tests - Start a local instance of etcd - Start a local instance of kube-apiserver - Start a local instance of kubelet - Run the test using the locally started processes - Output the test results to STDOUT - Stop kubelet, kube-apiserver, and etcd
Why Run tests Remotely? Tests will be run in a customized pristine environment. Closely mimics what will be done as pre- and post- submit testing performed by the project.
- join the googlegroup
- This provides read access to the node test images.
- Setup a Google Cloud Platform account and project with Google Compute Engine enabled
- Install and setup the gcloud sdk
- Verify the sdk is setup correctly by running
gcloud compute instances list and
gcloud compute images list --project kubernetes-node-e2e-images
make test-e2e-node REMOTE=true
- Build the Kubernetes source code
- Create a new GCE instance using the default test image
- Instance will be called test-e2e-node-containervm-v20160321-image
- Lookup the instance public ip address
- Copy a compressed archive file to the host containing the following binaries:
- e2e_node.test (this binary contains the actual tests to be run)
- Unzip the archive to a directory under /tmp/gcloud
- Run the tests using the
- Starts etcd, kube-apiserver, kubelet
- The ginkgo command is used because this supports more features than running the test binary directly
- Output the remote test results to STDOUT
scp the log files back to the local host under /tmp/_artifacts/e2e-node-containervm-v20160321-image
- Stop the processes on the remote host
- Leave the GCE instance running
Note: Subsequent tests run using the same image will reuse the existing host instead of deleting it and provisioning a new one. To delete the GCE instance after each test see DELETE_INSTANCE.
This is useful if you want to run tests against a host using a different OS distro or container runtime than provided by the default image.
List the available test images using gcloud.
make test-e2e-node LIST_IMAGES=true
This will output a list of the available images for the default image project.
make test-e2e-node REMOTE=true IMAGES="<comma-separated-list-images>"
This is useful if you have an host instance running already and want to run the tests there instead of on a new instance.
make test-e2e-node REMOTE=true HOSTS="<comma-separated-list-of-hostnames>"
This is useful if you want recreate the instance for each test run to trigger flakes related to starting the instance.
make test-e2e-node REMOTE=true DELETE_INSTANCES=true
This is useful if you want to manually inspect or debug the kubelet process run as part of the tests.
make test-e2e-node REMOTE=true CLEANUP=false
This is useful if you want to create your own host image in another project and use it for testing.
make test-e2e-node REMOTE=true IMAGE_PROJECT="<name-of-project-with-images>" IMAGES="<image-name>"
Setting up your own host image may require additional steps such as installing etcd or docker. See setup_host.sh for common steps to setup hosts to run node tests.
This is useful if you want to create instances using a different name so that you can run multiple copies of the test in parallel against different instances of the same image.
make test-e2e-node REMOTE=true INSTANCE_PREFIX="my-prefix"
To run tests matching a regex:
make test-e2e-node REMOTE=true FOCUS="<regex-to-match>"
To run tests NOT matching a regex:
make test-e2e-node REMOTE=true SKIP="<regex-to-match>"
This is useful if you are trying to debug a flaky test failure. This will cause ginkgo to continually run the tests until they fail. Note: this will only perform test setup once (e.g. creating the instance) and is less useful for catching flakes related creating the instance from an image.
make test-e2e-node REMOTE=true RUN_UNTIL_FAILURE=true
Running test in parallel can usually shorten the test duration. By default node
e2e test runs with
--nodes=8 (see ginkgo flag
–nodes). You can use the
PARALLELISM option to change the parallelism.
make test-e2e-node PARALLELISM=4 # run test with 4 parallel nodes make test-e2e-node PARALLELISM=1 # run test sequentially
Currently, kubenet is enabled by default for Remote execution
but disabled for Local execution. Note: kubenet is not supported for
local execution currently. This may cause network related test result to be
different for Local and Remote execution. So if you want to run network
related test, Remote execution is recommended.
To enable/disable kubenet:
# enable kubenet make test-e2e-node TEST_ARGS='--kubelet-flags="--network-plugin=kubenet --network-plugin-dir=/opt/cni/bin"' # disable kubenet make test-e2e-node TEST_ARGS='--kubelet-flags="--network-plugin= --network-plugin-dir="'
For testing with the QoS Cgroup Hierarchy enabled, you can pass –cgroups-per-qos flag as an argument into Ginkgo using TEST_ARGS
make test_e2e_node TEST_ARGS="--cgroups-per-qos=true"
The node e2e tests are run by the PR builder for each Pull Request and the results published at
the bottom of the comments section. To re-run just the node e2e tests from the PR builder add the comment
@k8s-bot node e2e test this issue: #<Flake-Issue-Number or IGNORE> and include a link to the test
failure logs if caused by a flake.
The PR builder runs tests against the images listed in jenkins-pull.properties
The post submit tests run against the images listed in jenkins-ci.properties