2020 Retrospective

Migrating my site to a new SSG and wanted to share a thought here. I wrote this two years ago and still use Ansible daily. That said a great Dockerfile or Packer are my go to’s now.

Intro

Writing code is fun, but what do you do when you get it working? Cron it on your computer and leave it on? You probably want to deploy it. Traditionally to a server, or maybe even a raspberry pi.

A month ago I wrote a program to automate some buying and trading so I didnt have to spend time on reddit. When I finished it, I saw it as a great opportunity to learn about containers. So here is a brief overview on how I used Ansible to deploy my code to Docker containers in an automated manner.

This makes a great entrance into container orchestration for the future when we want to deploy more complex applications with many pieces.

Well get started by going briefly over the python code we want to deploy, and then the steps we take to deploy it using Ansible-Container.

The code

Lets take a look at our code. The repo has three main parts.
All of which can be found on my github https://github.com/AnthonyLaiuppa/r_scrape

1. r_scrape.py - the Class doing all the heavy lifting for us
2. config.json - the configuration file with all the creds/variables we need to pass to r_scrape to make it work
3. run.py - the file that we actually execute to make the code go do the thing

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57

from __future__ import absolute_import, unicode_literals
from slackclient import SlackClient
import praw
import json 
import io

class rScrapeLogic(object):

	def __init__(self, mode=None):
		self.mode=mode

	def load_config(self, data):
		with io.open(data, mode='r', encoding='utf8') as config:
			self.config = json.loads(config.read())
		self.config['details']['keywords'] = [x.strip() for x in self.config['details']['keywords'].split(',')]
		print('Loaded Config successfully')
		return self.config

	def auth_reddit(self):
		try:
			self.reddit = praw.Reddit(
				client_id = self.config['reddit']['id'],
				client_secret = self.config['reddit']['secret'],
				user_agent = self.config['reddit']['user_agent'],
				username = self.config['reddit']['username']
			)
			print('Successfully authenticated to reddit api')
			return self.reddit
		except Exception as exc:
			print('{0} - Unable to auth to reddit, check your creds'.format(exc))	
			exit(0)

	def slack_it(self, message):	
		slack_client = SlackClient(self.config['slack']['token'])
		try:
			slack_client.rtm_connect()
			slack_client.api_call(
				"chat.postMessage", 
				channel=self.config['slack']['channel'], 
				text=message, 
				as_user=True)
		except Exception as exc:
			print('{0} - Unable to use slack, please check configs'.format(exc))

	def mon_subReddit(self):
		for submission in self.reddit.subreddit(self.config['details']['subreddit']).stream.submissions():
			try:
				print('going to check a submission {0}'.format(submission.url))
				self.check_submission(submission)
			except Exception as exc: 
				print('{0} - We are out of submissions waiting on more'.format(exc))

	def check_submission(self, submittal):
		for word in self.config['details']['keywords']:
			if word in submittal.selftext:
				message = '@channel We have a match - {0} - {1}'.format(word, submittal.url)
				self.slack_it(message)

Pretty succinct and clean minus, all the print statements I left from debugging.

Its a simple work flow to achieve what we want

Load config with credentials -> Authenticate to reddit -> Get posts -> Check for keywords -> Slack it if its got a keyword

Normally this is where I would put a picture of the results of the code running, but we want the container to run our code so lets lets the container give us the results.

Container Orchestration using Ansible

You can find all relevant code on my github if youd like to see it.

https://github.com/AnthonyLaiuppa/harbormaster
https://github.com/AnthonyLaiuppa/r_scrape

A container is essentially a stripped down virtual environment sitting ontop of the host kernel, we need to prepare it a little.

Pull the harbormaster repo, and then pull the r_scrape repo (reddit_scrape, couldve named it better)

Lets break down what we need to get this container ready to run the code.

1) Define the variables for the config.json by putting it in a vars file

2) Define the template for config.json so it receives the variables it needs to run the code.

This is really just following Jinja2 Documentation on templating for JSON, nothing unusual.

3) Fill out our tasks/main.yml so Ansible can do everything we need to prepare the Container.

The two main pieces that ansible-container needs to run are our tasks/main.yml and our container.yml

The tasks dictate what is going in the container and how, and the container.yml lays down the base rules and what should be done with it.

Lets take a look at our folder structure and then well look at the tasks we want to run.

We can see where all the relevant files sit after ansible-init as well as what weve added in the way of groupvars/, our template/ and our cloned source code of the app we want to deploy.

To begin with, what we seek to do is create a user on the system to run the code under. In production, ideally with least permission.

Copy our source over, install dependencies, remove the blank config from the repo clone, and then drop in our templated one.

From there our container.yml file tells docker that when it runs, it will to execute our run.py which in turns runs the flow describe earlier.

4) Build and run our container

Pieces are in place, lets build it ansible-container --vars-file group_vars/harbormaster.yml build --no-cache

As we can see our tasks executed cleanly, and our image is ready for use. Lets run it. ansible-container run

Summary

So what we achieved here was an incredibly quick flow of how we got our code running in a docker container, in an automated and manageable way.

If we were deploying to a production environment, we would tighten up some settings for security and use Kubernetes.

The ansible-container documentation goes over deploying to Kubernetes but since this is a pretty simple program I didnt feel the need to go that far.

Hopefully soon Ill be back with a more interesting application to distribute and manage with Kubernetes for you to read about.

As a surmise heres what we did

1. Develop a simple class with all of the methods we need to achieve our end goal
2. Use Ansible-Container to build our docker container
3. Use Ansible-Container to run our container

Really not all too complex, similar to using Vagrant/Ansible with provisioning VMs.

Keycaps are ran on limited runs in time restricted group buys, so if you miss the window of opportunity you can miss out on an awesome cap. Fortunately weve got some code keeping an eye out now.

Incase you were wondering what I was trying to obtain

I hope you enjoyed reading this, thanks for taking the time!