通过 Rally 进行 OpenStack Tempest 测试

Rally 基本介绍

Rally 是OpenStack社区推出开源测试工具，可用于对OpenStack各个组件进行性能测试。通过使用Rally组件，用户可完成OpenStack云计算平台的安装部署、功能验证、大规模负载测试(性能测试)、输出测试报告等一系列动作。
Rally 的概况和结构如下图所示：

Rally 主要包括三大部分：

• Deploy engine：这不是一个真的部署工具，它只是一个插件形式的东西，它可以和其他部署工具（比如 DevStack，Fuel，Anvil 等）一起工作来简化和统一部署流程。
• Verification：使用tempest验证已经部署的openstack云环境的功能。
• Benchmark engine：性能测试

Tempest 基本介绍

Tempest 是一个旨在为云计算平台 OpenStack 提供集成功能测试的开源项目，包含了 Openstack 基本组件（nova, keystone, glance, neutron, cinder 等）的 API 测试用例与场景。它是基于 unittest2 和 nose 建立的，灵活且易于扩展及维护，使得 OpenStack 相关测试效率得到大幅度提升。

安装 Rally

1. 安装依赖包

# yum install python-pip lsb_release gcc gmp-devel libffi-devel libxml2-devel libxslt-devel openssl-devel postgresql-devel python-devel RedHat-rpm-config

1. 安装 rally 最简单的方法就是使用下面的安装脚本

wget -q -O- https://raw.githubusercontent.com/openstack/rally/master/install_rally.sh | bash
# or using curl:
curl https://raw.githubusercontent.com/openstack/rally/master/install_rally.sh | bash

如果用普通用户执行脚本，Rally 会在 ==~/rally/== 下创建一个新的虚拟环境并安装在这里，使用 sqlite作为数据库后端。如果使用 root 用户执行脚本，Rally 会安装在系统路径，更多的安装选项，可以参考安装页面。

Rally 配置

1. 创建 openstack 环境变量文件，加入以下内容，注意修改用户名、密码、认证地址、region_name 等内容。

# vim admin-openrc

unset OS_SERVICE_TOKEN
    export OS_USERNAME=admin
    export OS_PASSWORD=admin
    export OS_AUTH_URL=http://192.168.3.222:5000/v3

export OS_PROJECT_NAME=admin
export OS_USER_DOMAIN_NAME=default
export OS_PROJECT_DOMAIN_NAME=Default
export OS_IDENTITY_API_VERSION=3
export OS_REGION_NAME=RegionOne

1. 加载环境变量

# . admin-openrc

1. 注册一个 Openstack deployment，注册成功后，将会默认使用这个 deployment，同时在主目录下会有一个新的目录出现：.rally。

# rally deployment create --fromenv --name=openstack

2017-07-31 15:44:12.509 20293 INFO rally.deployment.engines.existing [-] Save deployment 'openstack' (uuid=3403b234-76ae-4afb-9d96-49ef2d872069) with 'openstack' platform.
+--------------------------------------+---------------------+-----------+------------------+--------+
| uuid                                 | created_at          | name      | status           | active |
+--------------------------------------+---------------------+-----------+------------------+--------+
| 3403b234-76ae-4afb-9d96-49ef2d872069 | 2017-07-31T07:44:12 | openstack | deploy->finished |        |
+--------------------------------------+---------------------+-----------+------------------+--------+
Using deployment: 3403b234-76ae-4afb-9d96-49ef2d872069
~/.rally/openrc was updated

HINTS:

* To use standard OpenStack clients, set up your env by running:
    source ~/.rally/openrc
  OpenStack clients are now configured, e.g run:
    openstack image list

1. 检查一下刚注册的 deployment 是否存在。

# rally deployment list
+--------------------------------------+---------------------+-----------+------------------+--------+
| uuid                                 | created_at          | name      | status           | active |
+--------------------------------------+---------------------+-----------+------------------+--------+
| 3403b234-76ae-4afb-9d96-49ef2d872069 | 2017-07-31T07:44:12 | openstack | deploy->finished | *      |
+--------------------------------------+---------------------+-----------+------------------+--------+

1. 检查 deployment 是否可行

# rally deployment check

--------------------------------------------------------------------------------
Platform openstack:
--------------------------------------------------------------------------------

Available services:
+-------------+----------------+-----------+
| Service     | Service Type   | Status    |
+-------------+----------------+-----------+
| __unknown__ | alarming       | Available |
| __unknown__ | compute_legacy | Available |
| __unknown__ | event          | Available |
| __unknown__ | placement      | Available |
| __unknown__ | volumev2       | Available |
| __unknown__ | volumev3       | Available |
| cinder      | volume         | Available |
| glance      | image          | Available |
| gnocchi     | metric         | Available |
| keystone    | identity       | Available |
| neutron     | network        | Available |
| nova        | compute        | Available |
+-------------+----------------+-----------+

关于 service 显示 unknown 的问题可以参看以下文章：
rally deployment check is giving unknown under services
OpenStack Rally 性能测试

通过 Tempest verifier 验证云环境

1. 创建 Tempest verifier

#  rally verify create-verifier --type tempest --name tempest-verifier

1. 验证是否安装完成

# rally verify list-verifiers
+--------------------------------------+------------------+---------+-----------+---------------------+---------------------+-----------+---------+-------------+--------+
| UUID                                 | Name             | Type    | Namespace | Created at          | Updated at          | Status    | Version | System-wide | Active |
+--------------------------------------+------------------+---------+-----------+---------------------+---------------------+-----------+---------+-------------+--------+
| 4f4db99c-3930-442e-b592-bed5f428814e | tempest-verifier | tempest | openstack | 2017-07-31T05:24:09 | 2017-07-31T05:25:28 | installed | master  | False       | ✔      |
+--------------------------------------+------------------+---------+-----------+---------------------+---------------------+-----------+---------+-------------+--------+

1. 配置 Tempest verifier
   执行以下命令为当前部署配置 Tempest verifier

# rally verify configure-verifier

2017-07-31 15:56:33.940 20338 INFO rally.api [-] Configuring verifier 'tempest-verifier' (UUID=4f4db99c-3930-442e-b592-bed5f428814e) for deployment 'openstack' (UUID=3403b234-76ae-4afb-9d96-49ef2d872069).
2017-07-31 15:56:35.945 20338 INFO rally.api [-] Verifier 'tempest-verifier' (UUID=4f4db99c-3930-442e-b592-bed5f428814e) has been successfully configured for deployment 'openstack' (UUID=3403b234-76ae-4afb-9d96-49ef2d872069)!

查看配置信息

# rally verify configure-verifier --show

[DEFAULT]
debug = True
use_stderr = False
log_file = 

[auth]
use_dynamic_credentials = True
admin_username = admin
admin_password = admin
admin_project_name = admin
admin_domain_name = default
…………
[service_available]
cinder = True
glance = True
heat = False
ironic = False
neutron = True
nova = True
sahara = False
swift = False

[validation]
run_validation = True
image_ssh_user = cirros
connect_method = floating

[volume-feature-enabled]
bootable = True

开始验证

1. 执行以下命令开始验证

# rally verify start
2017-07-31 16:02:14.679 20417 INFO rally.api [-] Starting verification (UUID=ddca5b4b-03a9-49e4-8c91-1d53943ad10b) for deployment 'openstack' (UUID=3403b234-76ae-4afb-9d96-49ef2d872069) by verifier 'tempest-verifier' (UUID=4f4db99c-3930-442e-b592-bed5f428814e).
2017-07-31 16:02:25.381 20417 INFO tempest-verifier [-] {0} tempest.api.compute.admin.test_agents.AgentsAdminTestJSON.test_create_agent ... success [0.752s]
2017-07-31 16:02:25.972 20417 INFO tempest-verifier [-] {0} tempest.api.compute.admin.test_agents.AgentsAdminTestJSON.test_delete_agent ... success [0.588s]
2017-07-31 16:02:26.458 20417 INFO tempest-verifier [-] {0} tempest.api.compute.admin.test_agents.AgentsAdminTestJSON.test_list_agents ... success [0.486s]
2017-07-31 16:02:27.335 20417 INFO tempest-verifier [-] {0} tempest.api.compute.admin.test_agents.AgentsAdminTestJSON.test_list_agents_with_filter ... success [0.877s]
2017-07-31 16:02:27.975 20417 INFO tempest-verifier [-] {0} tempest.api.compute.admin.test_agents.AgentsAdminTestJSON.test_update_agent ... success [0.639s]
2017-07-31 16:02:37.491 20417 INFO tempest-verifier [-] {0} tempest.api.compute.admin.test_availability_zone.AZAdminV2TestJSON.test_get_availability_zone_list ... success [0.498s]
2017-07-31 16:02:38.042 20417 INFO tempest-verifier [-] {0} tempest.api.compute.admin.test_availability_zone.AZAdminV2TestJSON.test_get_availability_zone_list_detail ... success [0.551s]

默认情况下，以上命令会为当前部署执行完整的 tempest测试。

1. 可以使用 --pattern 选项只执行部分tempest测试

# rally verify start --pattern set=compute
2017-07-31 16:07:12.163 20459 INFO rally.api [-] Starting verification (UUID=4e36a2fb-5780-4db0-86bf-fe2b0ab92bf2) for deployment 'openstack' (UUID=3403b234-76ae-4afb-9d96-49ef2d872069) by verifier 'tempest-verifier' (UUID=4f4db99c-3930-442e-b592-bed5f428814e).
2017-07-31 16:07:17.189 20459 INFO tempest-verifier [-] {1} tempest.api.compute.admin.test_auto_allocate_network.AutoAllocateNetworkTest ... skip: The microversion range[2.37 - latest] of this test is out of the configuration range[None - None].
2017-07-31 16:07:21.786 20459 INFO tempest-verifier [-] {0} tempest.api.compute.admin.test_agents.AgentsAdminTestJSON.test_create_agent ... success [0.836s]
2017-07-31 16:07:23.170 20459 INFO tempest-verifier [-] {0} tempest.api.compute.admin.test_agents.AgentsAdminTestJSON.test_delete_agent ... success [1.382s]
2017-07-31 16:07:25.590 20459 INFO tempest-verifier [-] {0} tempest.api.compute.admin.test_agents.AgentsAdminTestJSON.test_list_agents ... success [2.422s]
2017-07-31 16:07:27.447 20459 INFO tempest-verifier [-] {0} tempest.api.compute.admin.test_agents.AgentsAdminTestJSON.test_list_agents_with_filter ... success [1.856s]
2017-07-31 16:07:28.135 20459 INFO tempest-verifier [-] {0} tempest.api.compute.admin.test_agents.AgentsAdminTestJSON.test_update_agent ... success [0.686s]

如 --pattern set=compute 选项，只会执行compute相关的测试。当前可供选择的测试内容有 full, smoke, compute, identity, image, network, object_storage, orchestration, volume, scenario

1. 用户可以使用正则表达式运行某些的测试集

# rally verify start --pattern tempest.api.compute.admin.test_flavors.FlavorsAdminTestJSON
2017-07-31 16:25:55.659 20502 INFO rally.api [-] Starting verification (UUID=84fce1ca-304b-4663-bba5-185f24d013a1) for deployment 'openstack' (UUID=3403b234-76ae-4afb-9d96-49ef2d872069) by verifier 'tempest-verifier' (UUID=4f4db99c-3930-442e-b592-bed5f428814e).
2017-07-31 16:26:03.792 20502 INFO tempest-verifier [-] {0} tempest.api.compute.admin.test_flavors.FlavorsAdminTestJSON.test_create_flavor_using_string_ram ... success [0.683s]
2017-07-31 16:26:04.703 20502 INFO tempest-verifier [-] {0} tempest.api.compute.admin.test_flavors.FlavorsAdminTestJSON.test_create_flavor_verify_entry_in_list_details ... success [0.910s]
2017-07-31 16:26:05.478 20502 INFO tempest-verifier [-] {0} tempest.api.compute.admin.test_flavors.FlavorsAdminTestJSON.test_create_flavor_with_int_id ... success [0.774s]
2017-07-31 16:26:06.230 20502 INFO tempest-verifier [-] {0} tempest.api.compute.admin.test_flavors.FlavorsAdminTestJSON.test_create_flavor_with_none_id ... success [0.750s]
2017-07-31 16:26:06.906 20502 INFO tempest-verifier [-] {0} tempest.api.compute.admin.test_flavors.FlavorsAdminTestJSON.test_create_flavor_with_uuid_id ... success [0.677s]
2017-07-31 16:26:08.224 20502 INFO tempest-verifier [-] {0} tempest.api.compute.admin.test_flavors.FlavorsAdminTestJSON.test_create_list_flavor_without_extra_data ... success [1.317s]
2017-07-31 16:26:09.264 20502 INFO tempest-verifier [-] {0} tempest.api.compute.admin.test_flavors.FlavorsAdminTestJSON.test_create_server_with_non_public_flavor ... success [1.038s]
2017-07-31 16:26:13.144 20502 INFO tempest-verifier [-] {0} tempest.api.compute.admin.test_flavors.FlavorsAdminTestJSON.test_is_public_string_variations ... success [3.873s]
2017-07-31 16:26:14.477 20502 INFO tempest-verifier [-] {0} tempest.api.compute.admin.test_flavors.FlavorsAdminTestJSON.test_list_non_public_flavor ... success [1.336s]
2017-07-31 16:26:15.548 20502 INFO tempest-verifier [-] {0} tempest.api.compute.admin.test_flavors.FlavorsAdminTestJSON.test_list_public_flavor_with_other_user ... success [1.067s]
2017-07-31 16:26:22.767 20502 INFO rally.api [-] Verification (UUID=84fce1ca-304b-4663-bba5-185f24d013a1) has been successfully finished for deployment 'openstack' (UUID=3403b234-76ae-4afb-9d96-49ef2d872069)!

======
Totals
======

Ran: 10 tests in 14.768 sec.
 - Success: 10
 - Skipped: 0
 - Expected failures: 0
 - Unexpected success: 0
 - Failures: 0

Using verification (UUID=84fce1ca-304b-4663-bba5-185f24d013a1) as the default verification for the future operations.

只会运行compute中和flavor相关的测试

1. 以这种方式，可以从某个目录或类运行测试，甚至可以运行单个测试

# rally verify start --pattern tempest.api.compute.admin.test_flavors.FlavorsAdminTestJSON.test_create_flavor_using_string_ram
2017-07-31 16:30:28.586 20533 INFO rally.api [-] Starting verification (UUID=181d37bd-d9a7-46fa-9311-ffe09d81e84c) for deployment 'openstack' (UUID=3403b234-76ae-4afb-9d96-49ef2d872069) by verifier 'tempest-verifier' (UUID=4f4db99c-3930-442e-b592-bed5f428814e).
2017-07-31 16:30:36.781 20533 INFO tempest-verifier [-] {0} tempest.api.compute.admin.test_flavors.FlavorsAdminTestJSON.test_create_flavor_using_string_ram ... success [0.772s]
2017-07-31 16:30:42.064 20533 INFO rally.api [-] Verification (UUID=181d37bd-d9a7-46fa-9311-ffe09d81e84c) has been successfully finished for deployment 'openstack' (UUID=3403b234-76ae-4afb-9d96-49ef2d872069)!

======
Totals
======

Ran: 1 tests in 2.734 sec.
 - Success: 1
 - Skipped: 0
 - Expected failures: 0
 - Unexpected success: 0
 - Failures: 0

Using verification (UUID=181d37bd-d9a7-46fa-9311-ffe09d81e84c) as the default verification for the future operations.

查看结果

我们可以报错结果为html、json等格式，一般保存为 html 格式，可以在浏览器中直观的查看

1. 获得 verify id

# rally verify list
+--------------------------------------+------+------------------+-----------------+---------------------+---------------------+----------+----------+
| UUID                                 | Tags | Verifier name    | Deployment name | Started at          | Finished at         | Duration | Status   |
+--------------------------------------+------+------------------+-----------------+---------------------+---------------------+----------+----------+
| db55c49c-9316-4353-94db-e0c777831157 | -    | tempest-verifier | openstack       | 2017-07-31T08:37:28 | 2017-07-31T14:40:46 | 6:03:18  | failed   |
+--------------------------------------+------+------------------+-----------------+---------------------+---------------------+----------+----------+

如果进行了多次测试会有多条劫夺，可以根据时间来区分，每次测试结束的时候都会提示本次测试的UUID

1. 导出为 html 文件

# rally verify report --uuid db55c49c-9316-4353-94db-e0c777831157 --type html --to export-name.html

1. 在浏览器中查看
   如图，可以显示所有的测试用例，错误的用例也会给出详细的错误信息