Openstack over k8s

Материал из Wiki
Перейти к: навигация, поиск

Комплексное задание на знание K8s/Helm/OpenStack на 8 рабочих часов (1 день)

* Install openstack(ceph as storage) on top of K8s(All-in-one-installation) using openstack-helm project 
* change Keystone token expiration time afterwards to 24 hours 
* deploy 3 VMs connected to each other using heat

TL; DR

  • Мне понадобилось примерно 13 рабочих часов что бы закончить задние

Хорошее:

  • Задание можно сделать за 8 часов (и даже быстрее)

Плохое

  • Практически невозможно сделать на ноутбуке без Linux.
  • Примерно половину времени потрачено на попытку "взлететь" напрямую на Mac OS и использовать в качестве кластера K8s уже имевшийся minikube

Это был явный фейл - как минимум чарты ceph не совместимы с миникубом никак (https://github.com/ceph/ceph-helm/issues/73), до остальных я не дошел.
Деплоить без скриптовой обвязки явно заняло бы больше чем 1 день (на самом деле если не срезать углы пользуясь скриптами то минимум неделя)

  • Когда я понял что задеплоить на миникуб за отведеннгое время не успеваю то решил настроить ВМку с убунтой и дальше работать с ней

Второй явный фейл (но не мой =) ) - то что задание требует минимум 8 гигов свободной памяти, а на самом деле даже на ВМке с 16-ю гигами и 8 ядрами все шевелилось очень медленно. (Человек с ноутом с 8 гигами не сделает это задание из-за недостатка памяти)
Как следствие - регулярные падения скриптов из-за таймаутов,
Отмечу так же что с не слишком быстрым интернетом я наступил на проблему, что Pull образов был медленным и скрипты не дожидались и падали по таймауту.
Хорошей идеей было бы скачать образы заранее, но об этом я подумал уже в середине процесса и тратить время на анализ какие образы нужны, не стал,

Решение

Логи деплоймента вынесены в отдельный раздел в конце что б не загромождать документ.

Создание ВМки с Убунтой

https://docs.openstack.org/openstack-helm/latest/install/developer/requirements-and-host-config.html

System Requirements¶
The recommended minimum system requirements for a full deployment are:

16GB of RAM
8 Cores
48GB HDD

VM-1.png


На этом этапе изначально я совершил 2 ошибки

  • Создал слишком маленькую (мало CPU Cores) машину
  • Не проверил пересечения сетей
 Warning
By default the Calico CNI will use 192.168.0.0/16 and Kubernetes services will use 10.96.0.0/16 as the CIDR for services. 
Check that these CIDRs are not in use on the development node before proceeding, or adjust as required.

Кстати, похоже что тут ошибка в маске в документации - реально используется маска /12
Немного отредактированный для удобства чтения вывод ps

root      5717  4.0  1.7 448168 292172 ?       Ssl  19:27   0:51      |   \_ kube-apiserver --feature-gates=MountPropagation=true,PodShareProcessNamespace=true 
                                                                                                      --service-node-port-range=1024-65535 
                                                                                                      --advertise-address=172.17.0.1 
                                                                                                      --service-cluster-ip-range=10.96.0.0/12

Подготовка

Если следовать инструкции и не пробовать ничего менять то никаких проблем на 18-й убунте не возникло.

Установка OpenStack

Если следовать инструкции то никаких проблем не возникает, за исключением таймаутов.
Насколько я смог выяснить - все скрипты делают корректную зачистку и потому перезапуск достаточно безопасен.
К сожалению я не сохранил список скриптов которые приходилось перезапускать

Проверка OpenStack

После окончания проверил самым простым способом - работает ли keystone:

root@openstack:~# export  OS_CLOUD=openstack_helm
root@openstack:~# openstack   token issue

На первый взгляд все PODы как минимум запустились

kubectl  -n openstack get pods
NAME                                                READY     STATUS      RESTARTS   AGE
ceph-ks-endpoints-hkj77                             0/3       Completed   0          3h
ceph-ks-service-l4wdx                               0/1       Completed   0          3h
ceph-openstack-config-ceph-ns-key-generator-z82mk   0/1       Completed   0          17h
ceph-rgw-66685f585d-st7dp                           1/1       Running     0          3h
ceph-rgw-storage-init-2vrpg                         0/1       Completed   0          3h
cinder-api-85df68f5d8-j6mqh                         1/1       Running     0          2h
cinder-backup-5f9598868-5kxxx                       1/1       Running     0          2h
cinder-backup-storage-init-g627m                    0/1       Completed   0          2h
cinder-bootstrap-r2295                              0/1       Completed   0          2h
cinder-db-init-nk7jm                                0/1       Completed   0          2h
cinder-db-sync-vlbcm                                0/1       Completed   0          2h
cinder-ks-endpoints-cnwgb                           0/9       Completed   0          2h
cinder-ks-service-6zs57                             0/3       Completed   0          2h
cinder-ks-user-bp8zb                                0/1       Completed   0          2h
cinder-rabbit-init-j97b7                            0/1       Completed   0          2h
cinder-scheduler-6bfcd6476d-r87hm                   1/1       Running     0          2h
cinder-storage-init-6ksjc                           0/1       Completed   0          2h
cinder-volume-5fccd4cc5-dpxqm                       1/1       Running     0          2h
cinder-volume-usage-audit-1549203300-25mkf          0/1       Completed   0          14m
cinder-volume-usage-audit-1549203600-hnh54          0/1       Completed   0          8m
cinder-volume-usage-audit-1549203900-v5t4w          0/1       Completed   0          4m
glance-api-745dc74457-42nwf                         1/1       Running     0          3h
glance-bootstrap-j5wt4                              0/1       Completed   0          3h
glance-db-init-lw97h                                0/1       Completed   0          3h
glance-db-sync-dbp5s                                0/1       Completed   0          3h
glance-ks-endpoints-gm5rw                           0/3       Completed   0          3h
glance-ks-service-64jfj                             0/1       Completed   0          3h
glance-ks-user-ftv9c                                0/1       Completed   0          3h
glance-rabbit-init-m7b7k                            0/1       Completed   0          3h
glance-registry-6cb86c767-2mkbx                     1/1       Running     0          3h
glance-storage-init-m29p4                           0/1       Completed   0          3h
heat-api-69db75bb6d-h24w9                           1/1       Running     0          3h
heat-bootstrap-v9642                                0/1       Completed   0          3h
heat-cfn-86896f7466-n5dnz                           1/1       Running     0          3h
heat-db-init-lfrsb                                  0/1       Completed   0          3h
heat-db-sync-wct2x                                  0/1       Completed   0          3h
heat-domain-ks-user-4fg65                           0/1       Completed   0          3h
heat-engine-6756c84fdd-44hzf                        1/1       Running     0          3h
heat-engine-cleaner-1549203300-s48sb                0/1       Completed   0          14m
heat-engine-cleaner-1549203600-gffn4                0/1       Completed   0          8m
heat-engine-cleaner-1549203900-6hwvj                0/1       Completed   0          4m
heat-ks-endpoints-wxjwp                             0/6       Completed   0          3h
heat-ks-service-v95sk                               0/2       Completed   0          3h
heat-ks-user-z6xhb                                  0/1       Completed   0          3h
heat-rabbit-init-77nzb                              0/1       Completed   0          3h
heat-trustee-ks-user-mwrf5                          0/1       Completed   0          3h
heat-trusts-7x7nt                                   0/1       Completed   0          3h
horizon-5877548d5d-27t8c                            1/1       Running     0          3h
horizon-db-init-jsjm5                               0/1       Completed   0          3h
horizon-db-sync-wxwpw                               0/1       Completed   0          3h
ingress-86cf786fd8-fbz8w                            1/1       Running     4          18h
ingress-error-pages-7f574d9cd7-b5kwh                1/1       Running     0          18h
keystone-api-f658f747c-q6w65                        1/1       Running     0          3h
keystone-bootstrap-ds8t5                            0/1       Completed   0          3h
keystone-credential-setup-hrp8t                     0/1       Completed   0          3h
keystone-db-init-dhgf2                              0/1       Completed   0          3h
keystone-db-sync-z8d5d                              0/1       Completed   0          3h
keystone-domain-manage-86b25                        0/1       Completed   0          3h
keystone-fernet-rotate-1549195200-xh9lv             0/1       Completed   0          2h
keystone-fernet-setup-txgc8                         0/1       Completed   0          3h
keystone-rabbit-init-jgkqz                          0/1       Completed   0          3h
libvirt-427lp                                       1/1       Running     0          2h
mariadb-ingress-5cff98cbfc-24vjg                    1/1       Running     0          17h
mariadb-ingress-5cff98cbfc-nqlhq                    1/1       Running     0          17h
mariadb-ingress-error-pages-5c89b57bc-twn7z         1/1       Running     0          17h
mariadb-server-0                                    1/1       Running     0          17h
memcached-memcached-6d48bd48bc-7kd84                1/1       Running     0          3h
neutron-db-init-rvf47                               0/1       Completed   0          2h
neutron-db-sync-6w7bn                               0/1       Completed   0          2h
neutron-dhcp-agent-default-znxhn                    1/1       Running     0          2h
neutron-ks-endpoints-47xs8                          0/3       Completed   1          2h
neutron-ks-service-sqtwg                            0/1       Completed   0          2h
neutron-ks-user-tpmrb                               0/1       Completed   0          2h
neutron-l3-agent-default-5nbsp                      1/1       Running     0          2h
neutron-metadata-agent-default-9ml6v                1/1       Running     0          2h
neutron-ovs-agent-default-mg8ln                     1/1       Running     0          2h
neutron-rabbit-init-sgnwm                           0/1       Completed   0          2h
neutron-server-9bdc765c9-bx6sf                      1/1       Running     0          2h
nova-api-metadata-78fb54c549-zcmxg                  1/1       Running     2          2h
nova-api-osapi-6c5c6dd4fc-7z5qq                     1/1       Running     0          2h
nova-bootstrap-hp6n4                                0/1       Completed   0          2h
nova-cell-setup-1549195200-v5bv8                    0/1       Completed   0          2h
nova-cell-setup-1549198800-6d8sm                    0/1       Completed   0          1h
nova-cell-setup-1549202400-c9vfz                    0/1       Completed   0          29m
nova-cell-setup-dfdzw                               0/1       Completed   0          2h
nova-compute-default-fmqtl                          1/1       Running     0          2h
nova-conductor-5b9956bffc-5ts7s                     1/1       Running     0          2h
nova-consoleauth-7f8dbb8865-lt5mr                   1/1       Running     0          2h
nova-db-init-hjp2p                                  0/3       Completed   0          2h
nova-db-sync-zn6px                                  0/1       Completed   0          2h
nova-ks-endpoints-ldzhz                             0/3       Completed   0          2h
nova-ks-service-c64tb                               0/1       Completed   0          2h
nova-ks-user-kjskm                                  0/1       Completed   0          2h
nova-novncproxy-6f485d9f4c-6m2n5                    1/1       Running     0          2h
nova-placement-api-587c888875-6cmmb                 1/1       Running     0          2h
nova-rabbit-init-t275g                              0/1       Completed   0          2h
nova-scheduler-69886c6fdf-hcwm6                     1/1       Running     0          2h
nova-service-cleaner-1549195200-7jw2d               0/1       Completed   1          2h
nova-service-cleaner-1549198800-pvckn               0/1       Completed   0          1h
nova-service-cleaner-1549202400-kqpxz               0/1       Completed   0          29m
openvswitch-db-nx579                                1/1       Running     0          2h
openvswitch-vswitchd-p4xj5                          1/1       Running     0          2h
placement-ks-endpoints-vt4pk                        0/3       Completed   0          2h
placement-ks-service-sw2b9                          0/1       Completed   0          2h
placement-ks-user-zv755                             0/1       Completed   0          2h
rabbitmq-rabbitmq-0                                 1/1       Running     0          4h
swift-ks-user-ktptt                                 0/1       Completed   0          3h

Доступ к Horizon

(настройки посмотрел в ингрессе, 10.255.57.3 - Адрес виртуальной машины

cat /etc/hosts
10.255.57.3 os horizon.openstack.svc.cluster.local

Horizon first login.png

Конфигурация KeyStone

В задании сказано:

change Keystone token expiration time afterwards to 24 hours 

Первое - проверим что там на самом деле

openstack token issue
+------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| Field      | Value                                                                                                                                                                                   |
+------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| expires    | 2019-02-04T00:25:34+0000                                                                                                                                                                |
| id         | gAAAAABcVt2-s8ugiwKaNQiA9djycTJ2CoDZ0sC176e54cjnE0RevPsXkgiZH0U5m_kNQlo0ctunA_TvD1tULyn0ckRkrO0Pxht1yT-cQ1TTidhkJR2sVojcXG3hiau0RMm0YOfoydDemyuvGMS7mwZ_Z2m9VtmJ-F83xQ8CwEfhItH6vRMzmGk |
| project_id | 2cb7f2c19a6f4e148bc3f9d0d0b7ed44                                                                                                                                                        |
| user_id    | 42068c166a3245208b5ac78965eab80b                                                                                                                                                        |
+------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+

Похоже что TTL=12h
Быстрое чтение документации ( https://docs.openstack.org/juno/config-reference/content/keystone-configuration-file.html ) привело меня к мысли что нужно менять секцию

[token]
expiration = 3600


Тут было принято решение сделать "быстро и грязно" и в реальном мире так скорее всего не выйдет,

1. Посчитать новое значение (вместо 24 вбил по ошибке 34)

bc
bc 1.07.1
Copyright 1991-1994, 1997, 1998, 2000, 2004, 2006, 2008, 2012-2017 Free Software Foundation, Inc.
This is free software with ABSOLUTELY NO WARRANTY.
For details type `warranty'.
34*60*60
122400
quit

2. Проверить что у нас один экземпляр keystone
Было б забавно если их было б больше и все выдавали токены с разным TTL

docker ps | grep keystone
41e785977105        16ec948e619f                                                     "/tmp/keystone-api.s…"   2 hours ago          Up 2 hours                              k8s_keystone-api_keystone-api-f658f747c-q6w65_openstack_8ca3a9ed-279f-11e9-a72e-080027da2b2f_0
6905400831ad        k8s.gcr.io/pause-amd64:3.1                                       "/pause"                 2 hours ago          Up 2 hours                              k8s_POD_keystone-api-f658f747c-q6w65_openstack_8ca3a9ed-279f-11e9-a72e-080027da2b2f_0

Тут конечно нужно kubectl exec ... но я решил срезать угол

docker exec -u root -ti 41e785977105  bash

Проверяю что запущено

ps -auxfw
USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
root       566  0.0  0.0  18236  3300 pts/0    Ss   12:34   0:00 bash
root       581  0.0  0.0  34428  2872 pts/0    R+   12:36   0:00  \_ ps -auxfw
keystone     1  0.0  1.1 263112 185104 ?       Ss   10:42   0:01 apache2 -DFOREGROUND
keystone    11  3.5  0.5 618912 95016 ?        Sl   10:42   4:03 (wsgi:k -DFOREGROUND
keystone   478  0.1  0.0 555276  9952 ?        Sl   12:23   0:00 apache2 -DFOREGROUND
keystone   506  0.2  0.0 555348  9956 ?        Sl   12:24   0:01 apache2 -DFOREGROUND

Соответствует ожиданиям .
Содержимое файла /etc/keystone.keystone.conf тоже - там как и предполагалось 12h

[token]
expiration = 43200

Поменять файл прямо на месте не удалось - что б уже все сделать не по правилам, изменил его снаружи на хосте

root@openstack:~# find /var -name keystone.conf
/var/lib/kubelet/pods/8ca3a9ed-279f-11e9-a72e-080027da2b2f/volumes/kubernetes.io~empty-dir/etckeystone/keystone.conf
/var/lib/kubelet/pods/8ca3a9ed-279f-11e9-a72e-080027da2b2f/volumes/kubernetes.io~secret/keystone-etc/..2019_02_03_12_37_10.041243569/keystone.conf
/var/lib/kubelet/pods/8ca3a9ed-279f-11e9-a72e-080027da2b2f/volumes/kubernetes.io~secret/keystone-etc/keystone.conf

и, возблагодарив разработчиков за то что keystone запущен под Apache (что позволило сделать релоад а не пересоздавать контейнер, а я не был уверен что знаю как это сделать правильно)

docker exec -u root -ti 41e785977105  bash
ps -auxfw
USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
root       566  0.0  0.0  18236  3300 pts/0    Ss   12:34   0:00 bash
root       581  0.0  0.0  34428  2872 pts/0    R+   12:36   0:00  \_ ps -auxfw
keystone     1  0.0  1.1 263112 185104 ?       Ss   10:42   0:01 apache2 -DFOREGROUND
keystone    11  3.5  0.5 618912 95016 ?        Sl   10:42   4:03 (wsgi:k -DFOREGROUND
keystone   478  0.1  0.0 555276  9952 ?        Sl   12:23   0:00 apache2 -DFOREGROUND
keystone   506  0.2  0.0 555348  9956 ?        Sl   12:24   0:01 apache2 -DFOREGROUND
root@keystone-api-f658f747c-q6w65:/etc/keystone# kill -HUP 1
root@keystone-api-f658f747c-q6w65:/etc/keystone# ps -auxfw
USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
root       566  0.0  0.0  18236  3300 pts/0    Ss   12:34   0:00 bash
root       583  0.0  0.0  34428  2888 pts/0    R+   12:36   0:00  \_ ps -auxfw
keystone     1  0.0  1.1 210588 183004 ?       Ss   10:42   0:01 apache2 -DFOREGROUND
keystone    11  3.5  0.0      0     0 ?        Z    10:42   4:03 [apache2] <defunct>
root@keystone-api-f658f747c-q6w65:/etc/keystone# ps -auxfw
USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
root       566  0.0  0.0  18236  3300 pts/0    Ss   12:34   0:00 bash
root       955  0.0  0.0  34428  2904 pts/0    R+   12:36   0:00  \_ ps -auxfw
keystone     1  0.0  1.1 263120 185124 ?       Ss   10:42   0:01 apache2 -DFOREGROUND
keystone   584 12.0  0.0 290680  8820 ?        Sl   12:36   0:00 (wsgi:k -DFOREGROUND
keystone   585 14.0  0.0 555188  9956 ?        Sl   12:36   0:00 apache2 -DFOREGROUND
keystone   586 14.0  0.0 555188  9956 ?        Sl   12:36   0:00 apache2 -DFOREGROUND
keystone   587 17.0  0.0 555188  9956 ?        Sl   12:36   0:00 apache2 -DFOREGROUND
keystone   588 13.0  0.0 555188  9956 ?        Sl   12:36   0:00 apache2 -DFOREGROUND
keystone   589 14.0  0.0 555188  9956 ?        Sl   12:36   0:00 apache2 -DFOREGROUND
keystone   590 10.0  0.0 555188 10020 ?        Sl   12:36   0:00 apache2 -DFOREGROUND
keystone   591 12.0  0.0 555188  9956 ?        Sl   12:36   0:00 apache2 -DFOREGROUND
keystone   592 10.0  0.0 555188  9956 ?        Sl   12:36   0:00 apache2 -DFOREGROUND
keystone   593 15.0  0.0 555188  9956 ?        Sl   12:36   0:00 apache2 -DFOREGROUND
keystone   594 14.0  0.0 265528  8572 ?        R    12:36   0:00 apache2 -DFOREGROUND
keystone   595 13.0  0.0 555188  9956 ?        Sl   12:36   0:00 apache2 -DFOREGROUND
keystone   596 11.0  0.0 266040  8832 ?        R    12:36   0:00 apache2 -DFOREGROUND
keystone   597 19.0  0.0 555188  9956 ?        Sl   12:36   0:00 apache2 -DFOREGROUND
keystone   598 14.0  0.0 555188  9956 ?        Sl   12:36   0:00 apache2 -DFOREGROUND
keystone   599 18.0  0.0 555188  9952 ?        Sl   12:36   0:00 apache2 -DFOREGROUND
keystone   600 11.0  0.0 265528  8376 ?        R    12:36   0:00 apache2 -DFOREGROUND

Проверяю применились ли изменения:

openstack token issue
+------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| Field      | Value                                                                                                                                                                                   |
+------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| expires    | 2019-02-04T22:37:10+0000                                                                                                                                                                |
| id         | gAAAAABcVuB2tQtAX56G1_kqJKeekpsWDJPTE19IMhWvNlGQqmDZQap9pgXQQkhQNMQNpR7Q6XR_w5_ngsx_l36vKXUND75uy4fimAbaLBDBdxxOzJqDRq4NLz4sEdTzLs2T3nyISwItLloOj-8sw7x1Pg2-9N-9afudv_jcYLVCq2luAImfRpY |
| project_id | 2cb7f2c19a6f4e148bc3f9d0d0b7ed44                                                                                                                                                        |
| user_id    | 42068c166a3245208b5ac78965eab80b                                                                                                                                                        |
+------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
root@openstack:/var/lib/kubelet/pods/8ca3a9ed-279f-11e9-a72e-080027da2b2f/volumes/kubernetes.io~secret/keystone-etc/..data# date
Sun Feb  3 12:37:18 UTC 2019

34 часа (из-за опечатки) но менять на 24 я уже не стал

Heat Deploy

deploy 3 VMs connected to each other using heat

Самая простая часть - все написано документации https://docs.openstack.org/openstack-helm/latest/install/developer/exercise-the-cloud.html
У меня с первого раза не создалась ВМка

[instance: 6aa5979e-1e03-4c8c-92bf-b1c1a43022ad] Instance failed to spawn
2019-02-03 13:20:56,466.466 21157 ERROR nova.compute.manager [instance: 6aa5979e-1e03-4c8c-92bf-b1c1a43022ad] Traceback (most recent call last):
2019-02-03 13:20:56,466.466 21157 ERROR nova.compute.manager [instance: 6aa5979e-1e03-4c8c-92bf-b1c1a43022ad]   File "/var/lib/openstack/local/lib/python2.7/site-packages/nova/compute/manager.py", line 2133, in _build_resources
2019-02-03 13:20:56,466.466 21157 ERROR nova.compute.manager [instance: 6aa5979e-1e03-4c8c-92bf-b1c1a43022ad]     yield resources
2019-02-03 13:20:56,466.466 21157 ERROR nova.compute.manager [instance: 6aa5979e-1e03-4c8c-92bf-b1c1a43022ad]   File "/var/lib/openstack/local/lib/python2.7/site-packages/nova/compute/manager.py", line 1939, in _build_and_run_instance
2019-02-03 13:20:56,466.466 21157 ERROR nova.compute.manager [instance: 6aa5979e-1e03-4c8c-92bf-b1c1a43022ad]     block_device_info=block_device_info)
2019-02-03 13:20:56,466.466 21157 ERROR nova.compute.manager [instance: 6aa5979e-1e03-4c8c-92bf-b1c1a43022ad]   File "/var/lib/openstack/local/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 2786, in spawn
2019-02-03 13:20:56,466.466 21157 ERROR nova.compute.manager [instance: 6aa5979e-1e03-4c8c-92bf-b1c1a43022ad]     block_device_info=block_device_info)
2019-02-03 13:20:56,466.466 21157 ERROR nova.compute.manager [instance: 6aa5979e-1e03-4c8c-92bf-b1c1a43022ad]   File "/var/lib/openstack/local/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 3193, in _create_image
2019-02-03 13:20:56,466.466 21157 ERROR nova.compute.manager [instance: 6aa5979e-1e03-4c8c-92bf-b1c1a43022ad]     fallback_from_host)
2019-02-03 13:20:56,466.466 21157 ERROR nova.compute.manager [instance: 6aa5979e-1e03-4c8c-92bf-b1c1a43022ad]   File "/var/lib/openstack/local/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 3309, in _create_and_inject_local_root
2019-02-03 13:20:56,466.466 21157 ERROR nova.compute.manager [instance: 6aa5979e-1e03-4c8c-92bf-b1c1a43022ad]     instance, size, fallback_from_host)
2019-02-03 13:20:56,466.466 21157 ERROR nova.compute.manager [instance: 6aa5979e-1e03-4c8c-92bf-b1c1a43022ad]   File "/var/lib/openstack/local/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 6953, in _try_fetch_image_cache
2019-02-03 13:20:56,466.466 21157 ERROR nova.compute.manager [instance: 6aa5979e-1e03-4c8c-92bf-b1c1a43022ad]     size=size)
2019-02-03 13:20:56,466.466 21157 ERROR nova.compute.manager [instance: 6aa5979e-1e03-4c8c-92bf-b1c1a43022ad]   File "/var/lib/openstack/local/lib/python2.7/site-packages/nova/virt/libvirt/imagebackend.py", line 242, in cache
2019-02-03 13:20:56,466.466 21157 ERROR nova.compute.manager [instance: 6aa5979e-1e03-4c8c-92bf-b1c1a43022ad]     *args, **kwargs)
2019-02-03 13:20:56,466.466 21157 ERROR nova.compute.manager [instance: 6aa5979e-1e03-4c8c-92bf-b1c1a43022ad]   File "/var/lib/openstack/local/lib/python2.7/site-packages/nova/virt/libvirt/imagebackend.py", line 584, in create_image
2019-02-03 13:20:56,466.466 21157 ERROR nova.compute.manager [instance: 6aa5979e-1e03-4c8c-92bf-b1c1a43022ad]     prepare_template(target=base, *args, **kwargs)
2019-02-03 13:20:56,466.466 21157 ERROR nova.compute.manager [instance: 6aa5979e-1e03-4c8c-92bf-b1c1a43022ad]   File "/var/lib/openstack/local/lib/python2.7/site-packages/oslo_concurrency/lockutils.py", line 271, in inner
2019-02-03 13:20:56,466.466 21157 ERROR nova.compute.manager [instance: 6aa5979e-1e03-4c8c-92bf-b1c1a43022ad]     return f(*args, **kwargs)
2019-02-03 13:20:56,466.466 21157 ERROR nova.compute.manager [instance: 6aa5979e-1e03-4c8c-92bf-b1c1a43022ad]   File "/var/lib/openstack/local/lib/python2.7/site-packages/nova/virt/libvirt/imagebackend.py", line 238, in fetch_func_sync
2019-02-03 13:20:56,466.466 21157 ERROR nova.compute.manager [instance: 6aa5979e-1e03-4c8c-92bf-b1c1a43022ad]     fetch_func(target=target, *args, **kwargs)
2019-02-03 13:20:56,466.466 21157 ERROR nova.compute.manager [instance: 6aa5979e-1e03-4c8c-92bf-b1c1a43022ad]   File "/var/lib/openstack/local/lib/python2.7/site-packages/nova/virt/libvirt/utils.py", line 458, in fetch_image
2019-02-03 13:20:56,466.466 21157 ERROR nova.compute.manager [instance: 6aa5979e-1e03-4c8c-92bf-b1c1a43022ad]     images.fetch_to_raw(context, image_id, target)
2019-02-03 13:20:56,466.466 21157 ERROR nova.compute.manager [instance: 6aa5979e-1e03-4c8c-92bf-b1c1a43022ad]   File "/var/lib/openstack/local/lib/python2.7/site-packages/nova/virt/images.py", line 132, in fetch_to_raw
2019-02-03 13:20:56,466.466 21157 ERROR nova.compute.manager [instance: 6aa5979e-1e03-4c8c-92bf-b1c1a43022ad]     fetch(context, image_href, path_tmp)
2019-02-03 13:20:56,466.466 21157 ERROR nova.compute.manager [instance: 6aa5979e-1e03-4c8c-92bf-b1c1a43022ad]   File "/var/lib/openstack/local/lib/python2.7/site-packages/nova/virt/images.py", line 123, in fetch
2019-02-03 13:20:56,466.466 21157 ERROR nova.compute.manager [instance: 6aa5979e-1e03-4c8c-92bf-b1c1a43022ad]     IMAGE_API.download(context, image_href, dest_path=path)
2019-02-03 13:20:56,466.466 21157 ERROR nova.compute.manager [instance: 6aa5979e-1e03-4c8c-92bf-b1c1a43022ad]   File "/var/lib/openstack/local/lib/python2.7/site-packages/nova/image/api.py", line 184, in download
2019-02-03 13:20:56,466.466 21157 ERROR nova.compute.manager [instance: 6aa5979e-1e03-4c8c-92bf-b1c1a43022ad]     dst_path=dest_path)
2019-02-03 13:20:56,466.466 21157 ERROR nova.compute.manager [instance: 6aa5979e-1e03-4c8c-92bf-b1c1a43022ad]   File "/var/lib/openstack/local/lib/python2.7/site-packages/nova/image/glance.py", line 533, in download
2019-02-03 13:20:56,466.466 21157 ERROR nova.compute.manager [instance: 6aa5979e-1e03-4c8c-92bf-b1c1a43022ad]     _reraise_translated_image_exception(image_id)
2019-02-03 13:20:56,466.466 21157 ERROR nova.compute.manager [instance: 6aa5979e-1e03-4c8c-92bf-b1c1a43022ad]   File "/var/lib/openstack/local/lib/python2.7/site-packages/nova/image/glance.py", line 1050, in _reraise_translated_image_exception
2019-02-03 13:20:56,466.466 21157 ERROR nova.compute.manager [instance: 6aa5979e-1e03-4c8c-92bf-b1c1a43022ad]     six.reraise(type(new_exc), new_exc, exc_trace)
2019-02-03 13:20:56,466.466 21157 ERROR nova.compute.manager [instance: 6aa5979e-1e03-4c8c-92bf-b1c1a43022ad]   File "/var/lib/openstack/local/lib/python2.7/site-packages/nova/image/glance.py", line 531, in download
2019-02-03 13:20:56,466.466 21157 ERROR nova.compute.manager [instance: 6aa5979e-1e03-4c8c-92bf-b1c1a43022ad]     image_chunks = self._client.call(context, 2, 'data', image_id)
2019-02-03 13:20:56,466.466 21157 ERROR nova.compute.manager [instance: 6aa5979e-1e03-4c8c-92bf-b1c1a43022ad]   File "/var/lib/openstack/local/lib/python2.7/site-packages/nova/image/glance.py", line 168, in call
2019-02-03 13:20:56,466.466 21157 ERROR nova.compute.manager [instance: 6aa5979e-1e03-4c8c-92bf-b1c1a43022ad]     result = getattr(controller, method)(*args, **kwargs)
2019-02-03 13:20:56,466.466 21157 ERROR nova.compute.manager [instance: 6aa5979e-1e03-4c8c-92bf-b1c1a43022ad]   File "/var/lib/openstack/local/lib/python2.7/site-packages/glanceclient/common/utils.py", line 535, in inner
2019-02-03 13:20:56,466.466 21157 ERROR nova.compute.manager [instance: 6aa5979e-1e03-4c8c-92bf-b1c1a43022ad]     return RequestIdProxy(wrapped(*args, **kwargs))
2019-02-03 13:20:56,466.466 21157 ERROR nova.compute.manager [instance: 6aa5979e-1e03-4c8c-92bf-b1c1a43022ad]   File "/var/lib/openstack/local/lib/python2.7/site-packages/glanceclient/v2/images.py", line 208, in data
2019-02-03 13:20:56,466.466 21157 ERROR nova.compute.manager [instance: 6aa5979e-1e03-4c8c-92bf-b1c1a43022ad]     resp, body = self.http_client.get(url)
2019-02-03 13:20:56,466.466 21157 ERROR nova.compute.manager [instance: 6aa5979e-1e03-4c8c-92bf-b1c1a43022ad]   File "/var/lib/openstack/local/lib/python2.7/site-packages/glanceclient/common/http.py", line 285, in get
2019-02-03 13:20:56,466.466 21157 ERROR nova.compute.manager [instance: 6aa5979e-1e03-4c8c-92bf-b1c1a43022ad]     return self._request('GET', url, **kwargs)
2019-02-03 13:20:56,466.466 21157 ERROR nova.compute.manager [instance: 6aa5979e-1e03-4c8c-92bf-b1c1a43022ad]   File "/var/lib/openstack/local/lib/python2.7/site-packages/glanceclient/common/http.py", line 277, in _request
2019-02-03 13:20:56,466.466 21157 ERROR nova.compute.manager [instance: 6aa5979e-1e03-4c8c-92bf-b1c1a43022ad]     resp, body_iter = self._handle_response(resp)
2019-02-03 13:20:56,466.466 21157 ERROR nova.compute.manager [instance: 6aa5979e-1e03-4c8c-92bf-b1c1a43022ad]   File "/var/lib/openstack/local/lib/python2.7/site-packages/glanceclient/common/http.py", line 107, in _handle_response
2019-02-03 13:20:56,466.466 21157 ERROR nova.compute.manager [instance: 6aa5979e-1e03-4c8c-92bf-b1c1a43022ad]     raise exc.from_response(resp, resp.content)
2019-02-03 13:20:56,466.466 21157 ERROR nova.compute.manager [instance: 6aa5979e-1e03-4c8c-92bf-b1c1a43022ad] HTTPInternalServerError: HTTPInternalServerError (HTTP 500)
2019-02-03 13:20:56,466.466 21157 ERROR nova.compute.manager [instance: 6aa5979e-1e03-4c8c-92bf-b1c1a43022ad]
2019-02-03 13:21:22,418.418 21157 INFO nova.compute.resource_tracker [req-cdb3800a-87ba-4ee9-88ad-e6914522a847 - - - - -] Final resource view: name=openstack phys_ram=16039MB used_ram=576MB phys_disk=48GB used_disk=1GB total_vcpus=8 used_vcpus=1 pci_stats=[]
2019-02-03 13:21:27,224.224 21157 INFO nova.compute.manager [req-c0895961-b263-4122-82cc-5267be0aad8f 42068c166a3245208b5ac78965eab80b 2cb7f2c19a6f4e148bc3f9d0d0b7ed44 - - -] [instance: 6aa5979e-1e03-4c8c-92bf-b1c1a43022ad] Terminating instance
2019-02-03 13:21:27,332.332 21157 INFO nova.virt.libvirt.driver [-] [instance: 6aa5979e-1e03-4c8c-92bf-b1c1a43022ad] Instance destroyed successfully.

Так как я подозревал что проблема в тормозах

2019-02-03 12:57:35,835.835 21157 WARNING nova.scheduler.client.report [req-cdb3800a-87ba-4ee9-88ad-e6914522a847 - - - - -] Failed to update inventory for resource provider 57daad8e-d831-4271-b3ef-332237d32b49: 503 503 Service Unavailable
The server is currently unavailable. Please try again at a later time.

то просто зачистил stack и закоментировал а скрипте создание сетей После чего VM-ка успешно создалась

 openstack server list
+--------------------------------------+----------------------------------------------+--------+------------------------------------------------------------------------+---------------------+----------------------------------------------+
| ID                                   | Name                                         | Status | Networks                                                               | Image               | Flavor                                       |
+--------------------------------------+----------------------------------------------+--------+------------------------------------------------------------------------+---------------------+----------------------------------------------+
| 155405cd-011a-42a2-93d7-3ed6eda250b2 | heat-basic-vm-deployment-server-ynxjzrycsd3z | ACTIVE | heat-basic-vm-deployment-private_net-tbltedh44qjv=10.0.0.4, 172.24.4.5 | Cirros 0.3.5 64-bit | heat-basic-vm-deployment-flavor-3kbmengg2bkm |
+--------------------------------------+----------------------------------------------+--------+------------------------------------------------------------------------+---------------------+----------------------------------------------+
root@openstack:~# openstack stack list
+--------------------------------------+-----------------------------+----------------------------------+-----------------+----------------------+--------------+
| ID                                   | Stack Name                  | Project                          | Stack Status    | Creation Time        | Updated Time |
+--------------------------------------+-----------------------------+----------------------------------+-----------------+----------------------+--------------+
| 1f90d25b-eb19-48cd-b623-cc0c7bccc28f | heat-vm-volume-attach       | 2cb7f2c19a6f4e148bc3f9d0d0b7ed44 | CREATE_COMPLETE | 2019-02-03T13:36:15Z | None         |
| 8d4ce486-ddb0-4826-b225-6b7dc4eef157 | heat-basic-vm-deployment    | 2cb7f2c19a6f4e148bc3f9d0d0b7ed44 | CREATE_COMPLETE | 2019-02-03T13:35:11Z | None         |
| d0aeea69-4639-4942-a905-ec30ed99aa47 | heat-subnet-pool-deployment | 2cb7f2c19a6f4e148bc3f9d0d0b7ed44 | CREATE_COMPLETE | 2019-02-03T13:16:29Z | None         |
| 688585e9-9b99-4ac7-bd04-9e7b874ec6c7 | heat-public-net-deployment  | 2cb7f2c19a6f4e148bc3f9d0d0b7ed44 | CREATE_COMPLETE | 2019-02-03T13:16:09Z | None         |
+--------------------------------------+-----------------------------+----------------------------------+-----------------+----------------------+--------------+

Теперь нужно создать три ВМки и проверить связь
Что б не делать это руками воспользовался тем же скриптом, дополнив его for I in $(seq 1 3); do

for I in $(seq 1 3); do
    openstack stack create --wait \
        --parameter public_net=${OSH_EXT_NET_NAME} \
        --parameter image="${IMAGE_NAME}" \
        --parameter ssh_key=${OSH_VM_KEY_STACK} \
        --parameter cidr=${OSH_PRIVATE_SUBNET} \
        --parameter dns_nameserver=${OSH_BR_EX_ADDR%/*} \
        -t ./tools/gate/files/heat-basic-vm-deployment.yaml \
        heat-basic-vm-deployment-${I}
...

Полученный результат

# openstack stack list
+--------------------------------------+-----------------------------+----------------------------------+-----------------+----------------------+--------------+
| ID                                   | Stack Name                  | Project                          | Stack Status    | Creation Time        | Updated Time |
+--------------------------------------+-----------------------------+----------------------------------+-----------------+----------------------+--------------+
| a6f1e35e-7536-4707-bdfa-b2885ab7cae2 | heat-vm-volume-attach-3     | 2cb7f2c19a6f4e148bc3f9d0d0b7ed44 | CREATE_COMPLETE | 2019-02-03T13:48:31Z | None         |
| ccb63a87-37f4-4355-b399-ef4abb43983b | heat-basic-vm-deployment-3  | 2cb7f2c19a6f4e148bc3f9d0d0b7ed44 | CREATE_COMPLETE | 2019-02-03T13:47:25Z | None         |
| 6d61b8de-80cd-4138-bfe2-8333a4b354ce | heat-vm-volume-attach-2     | 2cb7f2c19a6f4e148bc3f9d0d0b7ed44 | CREATE_COMPLETE | 2019-02-03T13:47:12Z | None         |
| 75c44f1c-d8da-422e-a027-f16b8458e224 | heat-basic-vm-deployment-2  | 2cb7f2c19a6f4e148bc3f9d0d0b7ed44 | CREATE_COMPLETE | 2019-02-03T13:46:08Z | None         |
| 95da63ac-9e20-4492-b3a6-fab74649bbf9 | heat-vm-volume-attach-1     | 2cb7f2c19a6f4e148bc3f9d0d0b7ed44 | CREATE_COMPLETE | 2019-02-03T13:45:54Z | None         |
| 447881bb-6c93-4b92-9765-578782ee2ef5 | heat-basic-vm-deployment-1  | 2cb7f2c19a6f4e148bc3f9d0d0b7ed44 | CREATE_COMPLETE | 2019-02-03T13:44:42Z | None         |
| 1f90d25b-eb19-48cd-b623-cc0c7bccc28f | heat-vm-volume-attach       | 2cb7f2c19a6f4e148bc3f9d0d0b7ed44 | CREATE_COMPLETE | 2019-02-03T13:36:15Z | None         |
| 8d4ce486-ddb0-4826-b225-6b7dc4eef157 | heat-basic-vm-deployment    | 2cb7f2c19a6f4e148bc3f9d0d0b7ed44 | CREATE_COMPLETE | 2019-02-03T13:35:11Z | None         |
| d0aeea69-4639-4942-a905-ec30ed99aa47 | heat-subnet-pool-deployment | 2cb7f2c19a6f4e148bc3f9d0d0b7ed44 | CREATE_COMPLETE | 2019-02-03T13:16:29Z | None         |
| 688585e9-9b99-4ac7-bd04-9e7b874ec6c7 | heat-public-net-deployment  | 2cb7f2c19a6f4e148bc3f9d0d0b7ed44 | CREATE_COMPLETE | 2019-02-03T13:16:09Z | None         |
+--------------------------------------+-----------------------------+----------------------------------+-----------------+----------------------+--------------+
openstack server list
+--------------------------------------+------------------------------------------------+--------+---------------------------------------------------------------------------+---------------------+------------------------------------------------+
| ID                                   | Name                                           | Status | Networks                                                                  | Image               | Flavor                                         |
+--------------------------------------+------------------------------------------------+--------+---------------------------------------------------------------------------+---------------------+------------------------------------------------+
| 412cbd8b-e4c1-46e8-b48c-065e9830bfa8 | heat-basic-vm-deployment-3-server-v5lwzoyotkwo | ACTIVE | heat-basic-vm-deployment-3-private_net-4unttrj2lq6z=10.0.0.6, 172.24.4.18 | Cirros 0.3.5 64-bit | heat-basic-vm-deployment-3-flavor-3gncn5vwfu6z |
| e7a4e42c-aa9c-47bc-ba7b-d229af1a2077 | heat-basic-vm-deployment-2-server-vhacv5jz7dnt | ACTIVE | heat-basic-vm-deployment-2-private_net-2gz44w5rjy7s=10.0.0.6, 172.24.4.11 | Cirros 0.3.5 64-bit | heat-basic-vm-deployment-2-flavor-hxr5eawiveg5 |
| 52886edd-be09-4ba1-aebd-5563f25f4f60 | heat-basic-vm-deployment-1-server-wk5lxhhnhhyn | ACTIVE | heat-basic-vm-deployment-1-private_net-hqx3dmohj3n5=10.0.0.5, 172.24.4.12 | Cirros 0.3.5 64-bit | heat-basic-vm-deployment-1-flavor-6aiokzvf4qaq |
| 155405cd-011a-42a2-93d7-3ed6eda250b2 | heat-basic-vm-deployment-server-ynxjzrycsd3z   | ACTIVE | heat-basic-vm-deployment-private_net-tbltedh44qjv=10.0.0.4, 172.24.4.5    | Cirros 0.3.5 64-bit | heat-basic-vm-deployment-flavor-3kbmengg2bkm   |
+--------------------------------------+------------------------------------------------+--------+---------------------------------------------------------------------------+---------------------+------------------------------------------------+

Проверка сети(22 порт точно открыт):

root@openstack:/etc/openstack# ssh -i /root/.ssh/osh_key cirros@172.24.4.18

$ nc 172.24.4.11 22
SSH-2.0-dropbear_2012.55
^Cpunt!

$ nc 172.24.4.12  22
SSH-2.0-dropbear_2012.55
^Cpunt!

$

Horizon with VMs

Horizon With VMs.png
Удивительно но заработала даже VNC консоль
Horizon VNC.png

Logs

MariaDB

+ ./tools/deployment/common/wait-for-pods.sh openstack
+ helm status mariadb
LAST DEPLOYED: Sun Feb  3 10:25:00 2019
NAMESPACE: openstack
STATUS: DEPLOYED

RESOURCES:
==> v1beta1/PodDisruptionBudget
NAME            MIN AVAILABLE  MAX UNAVAILABLE  ALLOWED DISRUPTIONS  AGE
mariadb-server  0              N/A              1                    13h

==> v1/ConfigMap
NAME                  DATA  AGE
mariadb-bin           5     13h
mariadb-etc           5     13h
mariadb-services-tcp  1     13h

==> v1/ServiceAccount
NAME                         SECRETS  AGE
mariadb-ingress-error-pages  1        13h
mariadb-ingress              1        13h
mariadb-mariadb              1        13h

==> v1beta1/RoleBinding
NAME                     AGE
mariadb-mariadb-ingress  13h
mariadb-ingress          13h
mariadb-mariadb          13h

==> v1/Service
NAME                         TYPE       CLUSTER-IP      EXTERNAL-IP  PORT(S)            AGE
mariadb-discovery            ClusterIP  None            <none>       3306/TCP,4567/TCP  13h
mariadb-ingress-error-pages  ClusterIP  None            <none>       80/TCP             13h
mariadb                      ClusterIP  10.104.164.168  <none>       3306/TCP           13h
mariadb-server               ClusterIP  10.107.255.234  <none>       3306/TCP           13h

==> v1/NetworkPolicy
NAME            POD-SELECTOR         AGE
mariadb-netpol  application=mariadb  13h

==> v1/Secret
NAME                      TYPE    DATA  AGE
mariadb-dbadmin-password  Opaque  1     13h
mariadb-secrets           Opaque  1     13h

==> v1beta1/Role
NAME                               AGE
mariadb-ingress                    13h
mariadb-openstack-mariadb-ingress  13h
mariadb-mariadb                    13h

==> v1/Deployment
NAME                         DESIRED  CURRENT  UP-TO-DATE  AVAILABLE  AGE
mariadb-ingress-error-pages  1        1        1           1          13h
mariadb-ingress              2        2        2           2          13h

==> v1/StatefulSet
NAME            DESIRED  CURRENT  AGE
mariadb-server  1        1        13h

==> v1/Pod(related)
NAME                                         READY  STATUS   RESTARTS  AGE
mariadb-ingress-error-pages-5c89b57bc-twn7z  1/1    Running  0         13h
mariadb-ingress-5cff98cbfc-24vjg             1/1    Running  0         13h
mariadb-ingress-5cff98cbfc-nqlhq             1/1    Running  0         13h
mariadb-server-0                             1/1    Running  0         13h


RabbitMQ

+ helm upgrade --install rabbitmq ../openstack-helm-infra/rabbitmq --namespace=openstack --values=/tmp/rabbitmq.yaml --set pod.replicas.server=1
Release "rabbitmq" does not exist. Installing it now.
NAME:   rabbitmq
LAST DEPLOYED: Sun Feb  3 10:27:01 2019
NAMESPACE: openstack
STATUS: DEPLOYED

RESOURCES:
==> v1/Service
NAME                 TYPE       CLUSTER-IP     EXTERNAL-IP  PORT(S)                       AGE
rabbitmq-dsv-7b1733  ClusterIP  None           <none>       5672/TCP,25672/TCP,15672/TCP  2s
rabbitmq-mgr-7b1733  ClusterIP  10.111.11.128  <none>       80/TCP,443/TCP                2s
rabbitmq             ClusterIP  10.108.248.80  <none>       5672/TCP,25672/TCP,15672/TCP  2s

==> v1/StatefulSet
NAME               DESIRED  CURRENT  AGE
rabbitmq-rabbitmq  1        1        2s

==> v1beta1/Ingress
NAME                 HOSTS                                                                                              ADDRESS  PORTS  AGE
rabbitmq-mgr-7b1733  rabbitmq-mgr-7b1733,rabbitmq-mgr-7b1733.openstack,rabbitmq-mgr-7b1733.openstack.svc.cluster.local  80       2s

==> v1/Pod(related)
NAME                 READY  STATUS   RESTARTS  AGE
rabbitmq-rabbitmq-0  0/1    Pending  0         2s

==> v1/ConfigMap
NAME                   DATA  AGE
rabbitmq-rabbitmq-bin  4     3s
rabbitmq-rabbitmq-etc  2     2s

==> v1/ServiceAccount
NAME               SECRETS  AGE
rabbitmq-test      1        2s
rabbitmq-rabbitmq  1        2s

==> v1/NetworkPolicy
NAME             POD-SELECTOR          AGE
rabbitmq-netpol  application=rabbitmq  2s

==> v1beta1/Role
NAME                              AGE
rabbitmq-openstack-rabbitmq-test  2s
rabbitmq-rabbitmq                 2s

==> v1beta1/RoleBinding
NAME                    AGE
rabbitmq-rabbitmq-test  2s
rabbitmq-rabbitmq       2s


+ ./tools/deployment/common/wait-for-pods.sh openstack
+ helm status rabbitmq
LAST DEPLOYED: Sun Feb  3 10:27:01 2019
NAMESPACE: openstack
STATUS: DEPLOYED

RESOURCES:
==> v1/Service
NAME                 TYPE       CLUSTER-IP     EXTERNAL-IP  PORT(S)                       AGE
rabbitmq-dsv-7b1733  ClusterIP  None           <none>       5672/TCP,25672/TCP,15672/TCP  2m21s
rabbitmq-mgr-7b1733  ClusterIP  10.111.11.128  <none>       80/TCP,443/TCP                2m21s
rabbitmq             ClusterIP  10.108.248.80  <none>       5672/TCP,25672/TCP,15672/TCP  2m21s

==> v1beta1/Ingress
NAME                 HOSTS                                                                                              ADDRESS  PORTS  AGE
rabbitmq-mgr-7b1733  rabbitmq-mgr-7b1733,rabbitmq-mgr-7b1733.openstack,rabbitmq-mgr-7b1733.openstack.svc.cluster.local  80       2m21s

==> v1/NetworkPolicy
NAME             POD-SELECTOR          AGE
rabbitmq-netpol  application=rabbitmq  2m21s

==> v1/ConfigMap
NAME                   DATA  AGE
rabbitmq-rabbitmq-bin  4     2m22s
rabbitmq-rabbitmq-etc  2     2m21s

==> v1/ServiceAccount
NAME               SECRETS  AGE
rabbitmq-test      1        2m21s
rabbitmq-rabbitmq  1        2m21s

==> v1beta1/RoleBinding
NAME                    AGE
rabbitmq-rabbitmq-test  2m21s
rabbitmq-rabbitmq       2m21s

==> v1beta1/Role
NAME                              AGE
rabbitmq-openstack-rabbitmq-test  2m21s
rabbitmq-rabbitmq                 2m21s

==> v1/StatefulSet
NAME               DESIRED  CURRENT  AGE
rabbitmq-rabbitmq  1        1        2m21s

==> v1/Pod(related)
NAME                 READY  STATUS   RESTARTS  AGE
rabbitmq-rabbitmq-0  1/1    Running  0         2m21s
root@openstack:~/mira/openstack-helm# echo $?
0

Memcached


+ helm upgrade --install memcached ../openstack-helm-infra/memcached --namespace=openstack --values=/tmp/memcached.yaml
Release "memcached" does not exist. Installing it now.
NAME:   memcached
LAST DEPLOYED: Sun Feb  3 10:30:32 2019
NAMESPACE: openstack
STATUS: DEPLOYED

RESOURCES:
==> v1/ConfigMap
NAME                     DATA  AGE
memcached-memcached-bin  1     3s

==> v1/ServiceAccount
NAME                 SECRETS  AGE
memcached-memcached  1        3s

==> v1/Service
NAME       TYPE       CLUSTER-IP     EXTERNAL-IP  PORT(S)    AGE
memcached  ClusterIP  10.96.106.159  <none>       11211/TCP  3s

==> v1/Deployment
NAME                 DESIRED  CURRENT  UP-TO-DATE  AVAILABLE  AGE
memcached-memcached  1        1        1           0          3s

==> v1/NetworkPolicy
NAME              POD-SELECTOR           AGE
memcached-netpol  application=memcached  3s

==> v1/Pod(related)
NAME                                  READY  STATUS    RESTARTS  AGE
memcached-memcached-6d48bd48bc-7kd84  0/1    Init:0/1  0         2s


+ ./tools/deployment/common/wait-for-pods.sh openstack
+ helm status memcached
LAST DEPLOYED: Sun Feb  3 10:30:32 2019
NAMESPACE: openstack
STATUS: DEPLOYED

RESOURCES:
==> v1/NetworkPolicy
NAME              POD-SELECTOR           AGE
memcached-netpol  application=memcached  78s

==> v1/Pod(related)
NAME                                  READY  STATUS   RESTARTS  AGE
memcached-memcached-6d48bd48bc-7kd84  1/1    Running  0         77s

==> v1/ConfigMap
NAME                     DATA  AGE
memcached-memcached-bin  1     78s

==> v1/ServiceAccount
NAME                 SECRETS  AGE
memcached-memcached  1        78s

==> v1/Service
NAME       TYPE       CLUSTER-IP     EXTERNAL-IP  PORT(S)    AGE
memcached  ClusterIP  10.96.106.159  <none>       11211/TCP  78s

==> v1/Deployment
NAME                 DESIRED  CURRENT  UP-TO-DATE  AVAILABLE  AGE
memcached-memcached  1        1        1           1          78s
root@openstack:~/mira/openstack-helm# echo $?
0

Keystone

+ ./tools/deployment/common/wait-for-pods.sh openstack
+ helm status keystone
LAST DEPLOYED: Sun Feb  3 10:36:19 2019
NAMESPACE: openstack
STATUS: DEPLOYED

RESOURCES:
==> v1/ServiceAccount
NAME                        SECRETS  AGE
keystone-credential-rotate  1        7m20s
keystone-fernet-rotate      1        7m20s
keystone-api                1        7m20s
keystone-bootstrap          1        7m20s
keystone-credential-setup   1        7m19s
keystone-db-init            1        7m19s
keystone-db-sync            1        7m19s
keystone-domain-manage      1        7m19s
keystone-fernet-setup       1        7m19s
keystone-rabbit-init        1        7m19s
keystone-test               1        7m19s

==> v1beta1/RoleBinding
NAME                                 AGE
keystone-keystone-credential-rotate  7m19s
keystone-credential-rotate           7m19s
keystone-fernet-rotate               7m19s
keystone-keystone-fernet-rotate      7m18s
keystone-keystone-api                7m18s
keystone-keystone-bootstrap          7m18s
keystone-credential-setup            7m18s
keystone-keystone-db-init            7m18s
keystone-keystone-db-sync            7m18s
keystone-keystone-domain-manage      7m18s
keystone-fernet-setup                7m18s
keystone-keystone-rabbit-init        7m18s
keystone-keystone-test               7m18s

==> v1/Service
NAME          TYPE       CLUSTER-IP      EXTERNAL-IP  PORT(S)         AGE
keystone-api  ClusterIP  10.110.158.186  <none>       5000/TCP        7m18s
keystone      ClusterIP  10.108.1.22     <none>       80/TCP,443/TCP  7m18s

==> v1/Deployment
NAME          DESIRED  CURRENT  UP-TO-DATE  AVAILABLE  AGE
keystone-api  1        1        1           1          7m18s

==> v1beta1/CronJob
NAME                        SCHEDULE      SUSPEND  ACTIVE  LAST SCHEDULE  AGE
keystone-credential-rotate  0 0 1 * *     False    0       <none>         7m18s
keystone-fernet-rotate      0 */12 * * *  False    0       <none>         7m18s

==> v1beta1/Ingress
NAME      HOSTS                                                             ADDRESS  PORTS  AGE
keystone  keystone,keystone.openstack,keystone.openstack.svc.cluster.local  80       7m18s

==> v1beta1/PodDisruptionBudget
NAME          MIN AVAILABLE  MAX UNAVAILABLE  ALLOWED DISRUPTIONS  AGE
keystone-api  0              N/A              1                    7m20s

==> v1/Secret
NAME                      TYPE    DATA  AGE
keystone-etc              Opaque  9     7m20s
keystone-credential-keys  Opaque  2     7m20s
keystone-db-admin         Opaque  1     7m20s
keystone-db-user          Opaque  1     7m20s
keystone-fernet-keys      Opaque  2     7m20s
keystone-keystone-admin   Opaque  8     7m20s
keystone-keystone-test    Opaque  8     7m20s
keystone-rabbitmq-admin   Opaque  1     7m20s
keystone-rabbitmq-user    Opaque  1     7m20s

==> v1/NetworkPolicy
NAME             POD-SELECTOR          AGE
keystone-netpol  application=keystone  7m18s

==> v1/Pod(related)
NAME                             READY  STATUS     RESTARTS  AGE
keystone-api-f658f747c-q6w65     1/1    Running    0         7m18s
keystone-bootstrap-ds8t5         0/1    Completed  0         7m18s
keystone-credential-setup-hrp8t  0/1    Completed  0         7m18s
keystone-db-init-dhgf2           0/1    Completed  0         7m18s
keystone-db-sync-z8d5d           0/1    Completed  0         7m18s
keystone-domain-manage-86b25     0/1    Completed  0         7m18s
keystone-fernet-setup-txgc8      0/1    Completed  0         7m18s
keystone-rabbit-init-jgkqz       0/1    Completed  0         7m18s

==> v1/Job
NAME                       COMPLETIONS  DURATION  AGE
keystone-bootstrap         1/1          7m13s     7m18s
keystone-credential-setup  1/1          2m6s      7m18s
keystone-db-init           1/1          3m46s     7m18s
keystone-db-sync           1/1          6m11s     7m18s
keystone-domain-manage     1/1          6m51s     7m18s
keystone-fernet-setup      1/1          3m52s     7m18s
keystone-rabbit-init       1/1          5m33s     7m18s

==> v1/ConfigMap
NAME          DATA  AGE
keystone-bin  13    7m20s

==> v1beta1/Role
NAME                                           AGE
keystone-openstack-keystone-credential-rotate  7m19s
keystone-credential-rotate                     7m19s
keystone-fernet-rotate                         7m19s
keystone-openstack-keystone-fernet-rotate      7m19s
keystone-openstack-keystone-api                7m19s
keystone-openstack-keystone-bootstrap          7m19s
keystone-credential-setup                      7m19s
keystone-openstack-keystone-db-init            7m19s
keystone-openstack-keystone-db-sync            7m19s
keystone-openstack-keystone-domain-manage      7m19s
keystone-fernet-setup                          7m19s
keystone-openstack-keystone-rabbit-init        7m19s
keystone-openstack-keystone-test               7m19s


+ export OS_CLOUD=openstack_helm
+ OS_CLOUD=openstack_helm
+ sleep 30
+ openstack endpoint list
+----------------------------------+-----------+--------------+--------------+---------+-----------+---------------------------------------------------------+
| ID                               | Region    | Service Name | Service Type | Enabled | Interface | URL                                                     |
+----------------------------------+-----------+--------------+--------------+---------+-----------+---------------------------------------------------------+
| 0f9f179d90a64e76ac65873826a4851e | RegionOne | keystone     | identity     | True    | internal  | http://keystone-api.openstack.svc.cluster.local:5000/v3 |
| 1ea5e2909c574c01bc815b96ba818db3 | RegionOne | keystone     | identity     | True    | public    | http://keystone.openstack.svc.cluster.local:80/v3       |
| 32e745bc02af4e5cb20830c83fc626e3 | RegionOne | keystone     | identity     | True    | admin     | http://keystone.openstack.svc.cluster.local:80/v3       |
+----------------------------------+-----------+--------------+--------------+---------+-----------+---------------------------------------------------------+
root@openstack:~/mira/openstack-helm# echo $?
0

Heat

+ :
+ helm upgrade --install heat ./heat --namespace=openstack --set manifests.network_policy=true
Release "heat" does not exist. Installing it now.
NAME:   heat
LAST DEPLOYED: Sun Feb  3 10:57:10 2019
NAMESPACE: openstack
STATUS: DEPLOYED

RESOURCES:
==> v1beta1/CronJob
NAME                 SCHEDULE     SUSPEND  ACTIVE  LAST SCHEDULE  AGE
heat-engine-cleaner  */5 * * * *  False    0       <none>         4s

==> v1/Secret
NAME                      TYPE    DATA  AGE
heat-etc                  Opaque  10    7s
heat-db-user              Opaque  1     7s
heat-db-admin             Opaque  1     7s
heat-keystone-user        Opaque  8     7s
heat-keystone-test        Opaque  8     7s
heat-keystone-stack-user  Opaque  5     7s
heat-keystone-trustee     Opaque  8     7s
heat-keystone-admin       Opaque  8     7s
heat-rabbitmq-admin       Opaque  1     7s
heat-rabbitmq-user        Opaque  1     7s

==> v1/ServiceAccount
NAME                  SECRETS  AGE
heat-engine-cleaner   1        7s
heat-api              1        7s
heat-cfn              1        7s
heat-engine           1        7s
heat-bootstrap        1        6s
heat-db-init          1        6s
heat-db-sync          1        6s
heat-ks-endpoints     1        6s
heat-ks-service       1        6s
heat-ks-user-domain   1        6s
heat-trustee-ks-user  1        6s
heat-ks-user          1        6s
heat-rabbit-init      1        6s
heat-trusts           1        6s
heat-test             1        6s

==> v1beta1/RoleBinding
NAME                       AGE
heat-heat-engine-cleaner   5s
heat-heat-api              5s
heat-heat-cfn              5s
heat-heat-engine           5s
heat-heat-db-init          5s
heat-heat-db-sync          5s
heat-heat-ks-endpoints     5s
heat-heat-ks-service       5s
heat-heat-ks-user-domain   5s
heat-heat-trustee-ks-user  5s
heat-heat-ks-user          5s
heat-heat-rabbit-init      5s
heat-heat-trusts           5s
heat-heat-test             5s

==> v1/Service
NAME            TYPE       CLUSTER-IP      EXTERNAL-IP  PORT(S)         AGE
heat-api        ClusterIP  10.107.126.110  <none>       8004/TCP        5s
heat-cfn        ClusterIP  10.103.165.157  <none>       8000/TCP        5s
heat            ClusterIP  10.106.167.63   <none>       80/TCP,443/TCP  5s
cloudformation  ClusterIP  10.107.173.42   <none>       80/TCP,443/TCP  5s

==> v1/Deployment
NAME         DESIRED  CURRENT  UP-TO-DATE  AVAILABLE  AGE
heat-api     1        1        1           0          5s
heat-cfn     1        1        1           0          5s
heat-engine  1        1        1           0          5s

==> v1/NetworkPolicy
NAME         POD-SELECTOR      AGE
heat-netpol  application=heat  4s

==> v1/Pod(related)
NAME                          READY  STATUS    RESTARTS  AGE
heat-api-69db75bb6d-h24w9     0/1    Init:0/1  0         5s
heat-cfn-86896f7466-n5dnz     0/1    Init:0/1  0         5s
heat-engine-6756c84fdd-44hzf  0/1    Init:0/1  0         5s
heat-bootstrap-v9642          0/1    Init:0/1  0         5s
heat-db-init-lfrsb            0/1    Pending   0         5s
heat-db-sync-wct2x            0/1    Init:0/1  0         5s
heat-ks-endpoints-wxjwp       0/6    Pending   0         5s
heat-ks-service-v95sk         0/2    Pending   0         5s
heat-domain-ks-user-4fg65     0/1    Pending   0         4s
heat-trustee-ks-user-mwrf5    0/1    Pending   0         4s
heat-ks-user-z6xhb            0/1    Pending   0         4s
heat-rabbit-init-77nzb        0/1    Pending   0         4s
heat-trusts-7x7nt             0/1    Pending   0         4s

==> v1beta1/PodDisruptionBudget
NAME      MIN AVAILABLE  MAX UNAVAILABLE  ALLOWED DISRUPTIONS  AGE
heat-api  0              N/A              0                    7s
heat-cfn  0              N/A              0                    7s

==> v1/ConfigMap
NAME      DATA  AGE
heat-bin  16    7s

==> v1beta1/Role
NAME                                 AGE
heat-openstack-heat-engine-cleaner   6s
heat-openstack-heat-api              6s
heat-openstack-heat-cfn              6s
heat-openstack-heat-engine           6s
heat-openstack-heat-db-init          6s
heat-openstack-heat-db-sync          6s
heat-openstack-heat-ks-endpoints     6s
heat-openstack-heat-ks-service       6s
heat-openstack-heat-ks-user-domain   6s
heat-openstack-heat-trustee-ks-user  6s
heat-openstack-heat-ks-user          6s
heat-openstack-heat-rabbit-init      6s
heat-openstack-heat-trusts           5s
heat-openstack-heat-test             5s

==> v1/Job
NAME                  COMPLETIONS  DURATION  AGE
heat-bootstrap        0/1          5s        5s
heat-db-init          0/1          4s        5s
heat-db-sync          0/1          5s        5s
heat-ks-endpoints     0/1          4s        5s
heat-ks-service       0/1          4s        5s
heat-domain-ks-user   0/1          4s        4s
heat-trustee-ks-user  0/1          4s        4s
heat-ks-user          0/1          4s        4s
heat-rabbit-init      0/1          4s        4s
heat-trusts           0/1          4s        4s

==> v1beta1/Ingress
NAME            HOSTS                                                                               ADDRESS  PORTS  AGE
heat            heat,heat.openstack,heat.openstack.svc.cluster.local                                80       4s
cloudformation  cloudformation,cloudformation.openstack,cloudformation.openstack.svc.cluster.local  80       4s


+ ./tools/deployment/common/wait-for-pods.sh openstack
+ export OS_CLOUD=openstack_helm
+ OS_CLOUD=openstack_helm
+ openstack service list
+----------------------------------+----------+----------------+
| ID                               | Name     | Type           |
+----------------------------------+----------+----------------+
| 5c354b75377944888ac1cc9a3a088808 | heat     | orchestration  |
| a55578c3ca8948d28055511c6a2e59bc | heat-cfn | cloudformation |
| fa7df0be3e99442d8fe42bda7519072f | keystone | identity       |
+----------------------------------+----------+----------------+
+ sleep 30
+ openstack orchestration service list
+------------------------------+-------------+--------------------------------------+-------------+--------+----------------------------+--------+
| Hostname                     | Binary      | Engine ID                            | Host        | Topic  | Updated At                 | Status |
+------------------------------+-------------+--------------------------------------+-------------+--------+----------------------------+--------+
| heat-engine-6756c84fdd-44hzf | heat-engine | 7a396564-9fce-43f9-aedd-3a48101925e8 | heat-engine | engine | 2019-02-03T11:03:45.000000 | up     |
+------------------------------+-------------+--------------------------------------+-------------+--------+----------------------------+--------+
root@openstack:~/mira/openstack-helm# echo $?
0

Horizon

+ helm status horizon
LAST DEPLOYED: Sun Feb  3 11:05:26 2019
NAMESPACE: openstack
STATUS: DEPLOYED

RESOURCES:
==> v1/Pod(related)
NAME                      READY  STATUS     RESTARTS  AGE
horizon-5877548d5d-27t8c  1/1    Running    0         6m22s
horizon-db-init-jsjm5     0/1    Completed  0         6m23s
horizon-db-sync-wxwpw     0/1    Completed  0         6m23s

==> v1/ConfigMap
NAME         DATA  AGE
horizon-bin  6     6m26s

==> v1/Service
NAME         TYPE       CLUSTER-IP      EXTERNAL-IP  PORT(S)         AGE
horizon      ClusterIP  10.111.206.119  <none>       80/TCP,443/TCP  6m24s
horizon-int  NodePort   10.107.139.114  <none>       80:31000/TCP    6m23s

==> v1/Job
NAME             COMPLETIONS  DURATION  AGE
horizon-db-init  1/1          37s       6m23s
horizon-db-sync  1/1          3m27s     6m23s

==> v1beta1/Role
NAME                               AGE
horizon-openstack-horizon          6m25s
horizon-openstack-horizon-db-init  6m24s
horizon-openstack-horizon-db-sync  6m24s

==> v1beta1/RoleBinding
NAME                     AGE
horizon-horizon          6m24s
horizon-horizon-db-init  6m24s
horizon-horizon-db-sync  6m24s

==> v1/Deployment
NAME     DESIRED  CURRENT  UP-TO-DATE  AVAILABLE  AGE
horizon  1        1        1           1          6m23s

==> v1beta1/Ingress
NAME     HOSTS                                                          ADDRESS  PORTS  AGE
horizon  horizon,horizon.openstack,horizon.openstack.svc.cluster.local  80       6m23s

==> v1/NetworkPolicy
NAME            POD-SELECTOR         AGE
horizon-netpol  application=horizon  6m23s

==> v1beta1/PodDisruptionBudget
NAME     MIN AVAILABLE  MAX UNAVAILABLE  ALLOWED DISRUPTIONS  AGE
horizon  0              N/A              1                    6m28s

==> v1/Secret
NAME              TYPE    DATA  AGE
horizon-etc       Opaque  10    6m27s
horizon-db-admin  Opaque  1     6m27s
horizon-db-user   Opaque  1     6m26s

==> v1/ServiceAccount
NAME             SECRETS  AGE
horizon          1        6m26s
horizon-db-init  1        6m26s
horizon-db-sync  1        6m26s


root@openstack:~/mira/openstack-helm# echo $?
0

Rados GW


+ helm upgrade --install radosgw-openstack ../openstack-helm-infra/ceph-rgw --namespace=openstack --values=/tmp/radosgw-openstack.yaml
Release "radosgw-openstack" does not exist. Installing it now.
NAME:   radosgw-openstack
E0203 11:14:02.662467   18336 portforward.go:303] error copying from remote stream to local connection: readfrom tcp4 127.0.0.1:33061->127.0.0.1:35352: write tcp4 127.0.0.1:33061->127.0.0.1:35352: write: broken pipe
LAST DEPLOYED: Sun Feb  3 11:14:02 2019
NAMESPACE: openstack
STATUS: DEPLOYED

RESOURCES:
==> v1/ServiceAccount
NAME                    SECRETS  AGE
ceph-rgw                1        0s
ceph-ks-endpoints       1        0s
ceph-ks-service         1        0s
swift-ks-user           1        0s
ceph-rgw-storage-init   1        0s
radosgw-openstack-test  1        0s

==> v1/Job
NAME                   COMPLETIONS  DURATION  AGE
ceph-ks-endpoints      0/1          0s        0s
ceph-ks-service        0/1          0s        0s
swift-ks-user          0/1          0s        0s
ceph-rgw-storage-init  0/1          0s        0s

==> v1beta1/RoleBinding
NAME                                      AGE
radosgw-openstack-ceph-rgw                0s
radosgw-openstack-ceph-ks-endpoints       0s
radosgw-openstack-ceph-ks-service         0s
radosgw-openstack-swift-ks-user           0s
ceph-rgw-storage-init                     0s
radosgw-openstack-radosgw-openstack-test  0s

==> v1/Service
NAME      TYPE       CLUSTER-IP    EXTERNAL-IP  PORT(S)         AGE
radosgw   ClusterIP  10.98.97.193  <none>       80/TCP,443/TCP  0s
ceph-rgw  ClusterIP  10.98.50.234  <none>       8088/TCP        0s

==> v1/Deployment
NAME      DESIRED  CURRENT  UP-TO-DATE  AVAILABLE  AGE
ceph-rgw  1        1        1           0          0s

==> v1beta1/Ingress
NAME     HOSTS                                                          ADDRESS  PORTS  AGE
radosgw  radosgw,radosgw.openstack,radosgw.openstack.svc.cluster.local  80       0s

==> v1/Pod(related)
NAME                         READY  STATUS    RESTARTS  AGE
ceph-rgw-66685f585d-st7dp    0/1    Init:0/3  0         0s
ceph-ks-endpoints-hkj77      0/3    Init:0/1  0         0s
ceph-ks-service-l4wdx        0/1    Init:0/1  0         0s
swift-ks-user-ktptt          0/1    Init:0/1  0         0s
ceph-rgw-storage-init-2vrpg  0/1    Init:0/2  0         0s

==> v1/Secret
NAME                    TYPE    DATA  AGE
ceph-keystone-user-rgw  Opaque  10    0s
ceph-keystone-user      Opaque  8     0s
ceph-keystone-admin     Opaque  8     0s
radosgw-s3-admin-creds  Opaque  3     0s

==> v1/ConfigMap
NAME                              DATA  AGE
ceph-rgw-bin-ks                   3     0s
ceph-rgw-bin                      7     0s
radosgw-openstack-ceph-templates  1     0s
ceph-rgw-etc                      1     0s

==> v1beta1/Role
NAME                                                AGE
radosgw-openstack-openstack-ceph-rgw                0s
radosgw-openstack-openstack-ceph-ks-endpoints       0s
radosgw-openstack-openstack-ceph-ks-service         0s
radosgw-openstack-openstack-swift-ks-user           0s
ceph-rgw-storage-init                               0s
radosgw-openstack-openstack-radosgw-openstack-test  0s


+ ./tools/deployment/common/wait-for-pods.sh openstack
+ helm status radosgw-openstack
LAST DEPLOYED: Sun Feb  3 11:14:02 2019
NAMESPACE: openstack
STATUS: DEPLOYED

RESOURCES:
==> v1beta1/Role
NAME                                                AGE
radosgw-openstack-openstack-ceph-rgw                3m54s
radosgw-openstack-openstack-ceph-ks-endpoints       3m54s
radosgw-openstack-openstack-ceph-ks-service         3m54s
radosgw-openstack-openstack-swift-ks-user           3m54s
ceph-rgw-storage-init                               3m54s
radosgw-openstack-openstack-radosgw-openstack-test  3m54s

==> v1beta1/RoleBinding
NAME                                      AGE
radosgw-openstack-ceph-rgw                3m54s
radosgw-openstack-ceph-ks-endpoints       3m54s
radosgw-openstack-ceph-ks-service         3m54s
radosgw-openstack-swift-ks-user           3m54s
ceph-rgw-storage-init                     3m54s
radosgw-openstack-radosgw-openstack-test  3m54s

==> v1/Service
NAME      TYPE       CLUSTER-IP    EXTERNAL-IP  PORT(S)         AGE
radosgw   ClusterIP  10.98.97.193  <none>       80/TCP,443/TCP  3m54s
ceph-rgw  ClusterIP  10.98.50.234  <none>       8088/TCP        3m54s

==> v1/Deployment
NAME      DESIRED  CURRENT  UP-TO-DATE  AVAILABLE  AGE
ceph-rgw  1        1        1           1          3m54s

==> v1/Pod(related)
NAME                         READY  STATUS     RESTARTS  AGE
ceph-rgw-66685f585d-st7dp    1/1    Running    0         3m54s
ceph-ks-endpoints-hkj77      0/3    Completed  0         3m54s
ceph-ks-service-l4wdx        0/1    Completed  0         3m54s
swift-ks-user-ktptt          0/1    Completed  0         3m54s
ceph-rgw-storage-init-2vrpg  0/1    Completed  0         3m54s

==> v1/Secret
NAME                    TYPE    DATA  AGE
ceph-keystone-user-rgw  Opaque  10    3m54s
ceph-keystone-user      Opaque  8     3m54s
ceph-keystone-admin     Opaque  8     3m54s
radosgw-s3-admin-creds  Opaque  3     3m54s

==> v1/ConfigMap
NAME                              DATA  AGE
ceph-rgw-bin-ks                   3     3m54s
ceph-rgw-bin                      7     3m54s
radosgw-openstack-ceph-templates  1     3m54s
ceph-rgw-etc                      1     3m54s

==> v1/ServiceAccount
NAME                    SECRETS  AGE
ceph-rgw                1        3m54s
ceph-ks-endpoints       1        3m54s
ceph-ks-service         1        3m54s
swift-ks-user           1        3m54s
ceph-rgw-storage-init   1        3m54s
radosgw-openstack-test  1        3m54s

==> v1/Job
NAME                   COMPLETIONS  DURATION  AGE
ceph-ks-endpoints      1/1          3m43s     3m54s
ceph-ks-service        1/1          3m22s     3m54s
swift-ks-user          1/1          3m50s     3m54s
ceph-rgw-storage-init  1/1          70s       3m54s

==> v1beta1/Ingress
NAME     HOSTS                                                          ADDRESS  PORTS  AGE
radosgw  radosgw,radosgw.openstack,radosgw.openstack.svc.cluster.local  80       3m54s


+ export OS_CLOUD=openstack_helm
+ OS_CLOUD=openstack_helm
+ sleep 30
+ openstack service list
+----------------------------------+----------+----------------+
| ID                               | Name     | Type           |
+----------------------------------+----------+----------------+
| 4dc87bba6ab94fd3b70e9d4493ef4e44 | swift    | object-store   |
| 5c354b75377944888ac1cc9a3a088808 | heat     | orchestration  |
| a55578c3ca8948d28055511c6a2e59bc | heat-cfn | cloudformation |
| fa7df0be3e99442d8fe42bda7519072f | keystone | identity       |
+----------------------------------+----------+----------------+
+ openstack container create mygreatcontainer
+--------------------------------------+------------------+-------------------------------------------------+
| account                              | container        | x-trans-id                                      |
+--------------------------------------+------------------+-------------------------------------------------+
| KEY_2cb7f2c19a6f4e148bc3f9d0d0b7ed44 | mygreatcontainer | tx000000000000000000018-005c56ce05-12f8-default |
+--------------------------------------+------------------+-------------------------------------------------+
+ curl -L -o /tmp/important-file.jpg https://imgflip.com/s/meme/Cute-Cat.jpg
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 35343  100 35343    0     0   168k      0 --:--:-- --:--:-- --:--:--  168k
+ openstack object create --name superimportantfile.jpg mygreatcontainer /tmp/important-file.jpg
+------------------------+------------------+----------------------------------+
| object                 | container        | etag                             |
+------------------------+------------------+----------------------------------+
| superimportantfile.jpg | mygreatcontainer | d09dbe3a95308bb4abd216885e7d1c34 |
+------------------------+------------------+----------------------------------+


OS_CLOUD=openstack_helm openstack object list mygreatcontainer
+------------------------+
| Name                   |
+------------------------+
| superimportantfile.jpg |
+------------------------+


Glance

+ export OS_CLOUD=openstack_helm
+ OS_CLOUD=openstack_helm
+ openstack service list
+----------------------------------+----------+----------------+
| ID                               | Name     | Type           |
+----------------------------------+----------+----------------+
| 0ef5d6114769472a896e7d5bfc2eb41a | glance   | image          |
| 4dc87bba6ab94fd3b70e9d4493ef4e44 | swift    | object-store   |
| 5c354b75377944888ac1cc9a3a088808 | heat     | orchestration  |
| a55578c3ca8948d28055511c6a2e59bc | heat-cfn | cloudformation |
| fa7df0be3e99442d8fe42bda7519072f | keystone | identity       |
+----------------------------------+----------+----------------+
+ sleep 30
+ openstack image list
+--------------------------------------+---------------------+--------+
| ID                                   | Name                | Status |
+--------------------------------------+---------------------+--------+
| ccfed5c7-b652-4dbc-8fc9-6dc4fec4985c | Cirros 0.3.5 64-bit | active |
+--------------------------------------+---------------------+--------+
+ openstack image show 'Cirros 0.3.5 64-bit'
+------------------+------------------------------------------------------+
| Field            | Value                                                |
+------------------+------------------------------------------------------+
| checksum         | f8ab98ff5e73ebab884d80c9dc9c7290                     |
| container_format | bare                                                 |
| created_at       | 2019-02-03T11:26:45Z                                 |
| disk_format      | qcow2                                                |
| file             | /v2/images/ccfed5c7-b652-4dbc-8fc9-6dc4fec4985c/file |
| id               | ccfed5c7-b652-4dbc-8fc9-6dc4fec4985c                 |
| min_disk         | 1                                                    |
| min_ram          | 0                                                    |
| name             | Cirros 0.3.5 64-bit                                  |
| owner            | 2cb7f2c19a6f4e148bc3f9d0d0b7ed44                     |
| properties       | hypervisor_type='qemu', os_distro='cirros'           |
| protected        | False                                                |
| schema           | /v2/schemas/image                                    |
| size             | 13267968                                             |
| status           | active                                               |
| tags             |                                                      |
| updated_at       | 2019-02-03T11:26:48Z                                 |
| virtual_size     | None                                                 |
| visibility       | private                                              |
+------------------+------------------------------------------------------+

Cinder

+ ./tools/deployment/common/wait-for-pods.sh openstack
+ export OS_CLOUD=openstack_helm
+ OS_CLOUD=openstack_helm
+ openstack service list
+----------------------------------+----------+----------------+
| ID                               | Name     | Type           |
+----------------------------------+----------+----------------+
| 0ef5d6114769472a896e7d5bfc2eb41a | glance   | image          |
| 151bcbb92c854322ae154447cc58662b | cinderv2 | volumev2       |
| 4dc87bba6ab94fd3b70e9d4493ef4e44 | swift    | object-store   |
| 5c354b75377944888ac1cc9a3a088808 | heat     | orchestration  |
| a55578c3ca8948d28055511c6a2e59bc | heat-cfn | cloudformation |
| f25bb35c9bf147e4b0d10487c8e8eeaf | cinderv3 | volumev3       |
| f32899524d4b46248ca82d317748bbfd | cinder   | volume         |
| fa7df0be3e99442d8fe42bda7519072f | keystone | identity       |
+----------------------------------+----------+----------------+
+ sleep 30
+ openstack volume type list
+--------------------------------------+------+-----------+
| ID                                   | Name | Is Public |
+--------------------------------------+------+-----------+
| 25ae326e-f840-4cb9-802e-4646dd237cad | rbd1 | True      |
+--------------------------------------+------+-----------+
root@openstack:~/mira/openstack-helm# echo $?
0

OpenVSwitch=

+ ./tools/deployment/common/wait-for-pods.sh openstack
+ helm status openvswitch
LAST DEPLOYED: Sun Feb  3 11:43:52 2019
NAMESPACE: openstack
STATUS: DEPLOYED

RESOURCES:
==> v1/ConfigMap
NAME             DATA  AGE
openvswitch-bin  3     113s

==> v1/ServiceAccount
NAME                  SECRETS  AGE
openvswitch-db        1        113s
openvswitch-vswitchd  1        113s

==> v1/DaemonSet
NAME                  DESIRED  CURRENT  READY  UP-TO-DATE  AVAILABLE  NODE SELECTOR        AGE
openvswitch-db        1        1        1      1           1          openvswitch=enabled  113s
openvswitch-vswitchd  1        1        1      1           1          openvswitch=enabled  113s

==> v1/NetworkPolicy
NAME                POD-SELECTOR             AGE
openvswitch-netpol  application=openvswitch  113s

==> v1/Pod(related)
NAME                        READY  STATUS   RESTARTS  AGE
openvswitch-db-nx579        1/1    Running  0         113s
openvswitch-vswitchd-p4xj5  1/1    Running  0         113s

root@openstack:~/mira/openstack-helm# echo $?
0

LibVirt

+ helm upgrade --install libvirt ../openstack-helm-infra/libvirt --namespace=openstack --set manifests.network_policy=true --values=/tmp/libvirt.yaml
Release "libvirt" does not exist. Installing it now.
NAME:   libvirt
LAST DEPLOYED: Sun Feb  3 11:46:46 2019
NAMESPACE: openstack
STATUS: DEPLOYED

RESOURCES:
==> v1/Pod(related)
NAME           READY  STATUS    RESTARTS  AGE
libvirt-427lp  0/1    Init:0/3  0         0s

==> v1/ConfigMap
NAME         DATA  AGE
libvirt-bin  3     0s
libvirt-etc  2     0s

==> v1/ServiceAccount
NAME     SECRETS  AGE
libvirt  1        0s

==> v1beta1/Role
NAME                       AGE
libvirt-openstack-libvirt  0s

==> v1beta1/RoleBinding
NAME             AGE
libvirt-libvirt  0s

==> v1/DaemonSet
NAME     DESIRED  CURRENT  READY  UP-TO-DATE  AVAILABLE  NODE SELECTOR                   AGE
libvirt  1        1        0      1           0          openstack-compute-node=enabled  0s

==> v1/NetworkPolicy
NAME            POD-SELECTOR         AGE
libvirt-netpol  application=libvirt  0s


+ helm status libvirt
LAST DEPLOYED: Sun Feb  3 11:46:46 2019
NAMESPACE: openstack
STATUS: DEPLOYED

RESOURCES:
==> v1beta1/Role
NAME                       AGE
libvirt-openstack-libvirt  1s

==> v1beta1/RoleBinding
NAME             AGE
libvirt-libvirt  1s

==> v1/DaemonSet
NAME     DESIRED  CURRENT  READY  UP-TO-DATE  AVAILABLE  NODE SELECTOR                   AGE
libvirt  1        1        0      1           0          openstack-compute-node=enabled  1s

==> v1/NetworkPolicy
NAME            POD-SELECTOR         AGE
libvirt-netpol  application=libvirt  1s

==> v1/Pod(related)
NAME           READY  STATUS    RESTARTS  AGE
libvirt-427lp  0/1    Init:0/3  0         1s

==> v1/ConfigMap
NAME         DATA  AGE
libvirt-bin  3     1s
libvirt-etc  2     1s

==> v1/ServiceAccount
NAME     SECRETS  AGE
libvirt  1        1s
root@openstack:~/mira/openstack-helm# echo $?
0