Configuring rabbitmq access in an Ansible All In One environment

I'm working on implementing an OpenStack Ansible Searchlight role.

I've been having some problems getting access to rabbitmq to work.

It turns out that rabbitmq is a bit sucky, but add in some cluelessness on my part and you have a recipe for frustration.

For the last couple of days, I had been stuck on getting the searchlight-listener to actually connect to rabbitmq via oslo.messaging.

I was seeing lots of lines in /var/log/searchlight/searchlight-lister.log along the lines of:

2016-11-02 00:05:49.096 8871 ERROR oslo.messaging._drivers.impl_rabbit [-] [462d7cb1-f1c4-4f76-8628-57d5265fd469] AMQP server on 172.29.236.234:5671 is unreachable: [Errno 104] Connection reset by peer. Trying again in 32 seconds. Client port: None 
2016-11-02 00:06:21.137 8871 ERROR oslo.messaging._drivers.impl_rabbit [-] [462d7cb1-f1c4-4f76-8628-57d5265fd469] AMQP server on 172.29.236.234:5671 is unreachable: [Errno 104] Connection reset by peer. Trying again in 32 seconds. Client port: None 
2016-11-02 00:06:53.172 8871 ERROR oslo.messaging._drivers.impl_rabbit [-] [462d7cb1-f1c4-4f76-8628-57d5265fd469] AMQP server on 172.29.236.234:5671 is unreachable: [Errno 104] Connection reset by peer. Trying again in 32 seconds. Client port: None 
2016-11-02 00:07:25.209 8871 ERROR oslo.messaging._drivers.impl_rabbit [-] [462d7cb1-f1c4-4f76-8628-57d5265fd469] AMQP server on 172.29.236.234:5671 is unreachable: [Errno 104] Connection reset by peer. Trying again in 32 seconds. Client port: None 
2016-11-02 00:07:57.246 8871 ERROR oslo.messaging._drivers.impl_rabbit [-] [462d7cb1-f1c4-4f76-8628-57d5265fd469] AMQP server on 172.29.236.234:5671 is unreachable: [Errno 104] Connection reset by peer. Trying again in 32 seconds. Client port: None 

I looked at all sorts of things - permissions, user_secrets, networking, SSL, certificates all to no avail.

It turns out that all I actually needed to do was add the following to searchlight's configuration:

[oslo_messaging_rabbit] 
rabbit_use_ssl = {{ searchlight_rabbitmq_use_ssl }}

I had been setting:
 searchlight_rabbitmq_use_ssl: {{ searchlight_rabbitmq_use_ssl }} in the [DEFAULT] section, but it looks like that was just magical thinking on my part.

So, in the hopes that if I ever do this again having this public admission of an id-ten-t problem will be worth the time it may save me, here's a blog post.

Somehow I have a feeling that the majority of my blog posts will be me admitting to doing something stupid.

How To Boot from a Volume on Rackspace Public Cloud using pyrax or the rack cli

For reasons too long to go into I recently had a need to create a VM that booted from a volume on the Rackspace Public Cloud.

This is fairly easy to achieve from the web interface - a couple of mouse clicks and you're done.

It's also possible from the rack cli tool. Something like:

rack servers instance create --name=test-yy \
  --flavor-id=io1-15 \
  --block-device="source-type=image,source-id=<uuid>,destination-type=volume,volume-size=150"

will do the trick.

None of these perfectly acceptable methods were quite what I wanted so I ended up poking at pyrax and the nova client until I got a solution I could use from python.

 1 import pyrax
 2
 3 def create_server(name, keypair):
 4     pyrax.set_credential_file("/path/to/pyraxcreds")
 5     nova = pyrax.cloudservers
 6     bdm = {
 7         'source_type': 'image',
 8         'uuid': '1d3ea64f-1ead-4042-8cb6-8ceb523b6149',
 9         'destination_type': 'volume',
10         'volume_size': '150',
11         'boot_index': '0'
12     }
13     
14     my_server = nova.servers.create(
15         name=name,
16         image=None,
17         flavor='io1-15',
18         key_name=keypair,
19         block_device_mapping_v2=[bdm]
20     )
21     
22     print(my_server.status)
23 
24 
25 def main():
26     create_server('test-xx', 'neill')
27 
28 main()

Taking this apart, the important bits are:

Line 4: Before you can use pyrax you need to provide some configuration.  Most of it can live in ~/.pyrax.cfg (see for an explanation of what belongs in this file), but your credentials must live elsewhere and you must tell pyrax where to find them before using pyrax to do anything. The credentials file needs to supply a username and for the Rackspace public cloud an API key.

Line 7-11 are where the magic really happens.  This sets up a nova block device mapping to tell it how to boot the VM. We are booting from an image with the UUID 1d3e...6149 (which specifies an Ubuntu Xenial image in the Rackspace public cloud), the boot destination is a volume 150GB in size, and it has boot index zero (meaning it is the first boot device).

Line 16: Even though we are supplying a boot device mapping nova requires an image parameter to be passed in here, but it will ignore the actual value.

Line 17: is the flavor-id for the VM we are creating.  This will vary by cloud.  On the Rackspace public cloud you can get a list of flavors by running:

rack servers flavor list

Line 18: The name of a keypair known to the Rackspace public cloud so you don't need to remember the root password.

Line 19: The nova client expects the block device mapping to be a list (presumably you can have more than one block device, but I only need the one).

Note: as presented here this code will spew InsecureRequestWarning exceptions from the requests library. To ignore these (probably not wise) you can add the following two lines to the code:

from requests.packages.urllib3.exceptions import InsecureRequestWarning
requests.packages.urllib3.disable_warnings(InsecureRequestWarning)

Hope this saves the next person some time!