Monthly Archives: February 2010

Continuous Integration with Integrity

What is he banging on about this time?

Having a Continuous Integration system in place is a central practice of a functioning agile development team.

Our old CI server was bardy, supa dupa bardy, and over complicated. We were using cruisecontrol.rb, it did the trick but we were using java to run our selenium tests, which was a bit of a pain, so we decided to use Integrity to handle it all instead. ( And dropped the selenium tests for the time being and just use cucumber and rspec. )

So you’ve built a bog standard Ubuntu Server build with ruby and apache with passenger installed, as well as all the normal security, monitoring and maintenance software loaded and configured. (Hopefully we’ll get an article up soon on how to do all this, without giving away too many secrets of course :-))

You’ll also need to install all the software on the system that you’ll need to run whatever tests you want to run in every project that needs to be tested. Nearly all the gems will be bundled, so they’re not so much of a worry, but things like imagemagick and relevant development libraries and system tools need to be installed.

Install Integrity

Let’s grab the code, install it and create the sqlite db

$ gem install bundler
$ git clone git://github.com/integrity/integrity
$ cd integrity
$ git checkout -b deploy v0.2.3
$ gem bundle --only default
$ ./bin/rake db

Configure Integrity

init.rb

uncomment this line(we’re going to use campfire notifier)

require "integrity/notifier/campfire"

and change c.base_url to be the url of your integrity server

c.base_url     "myintegrityserver.com"

config.ru

add this block prior to running the app to add some security to the site

use Rack::Auth::Basic do |user, pass|
  user == "admin" && pass == "secret"
end

Gemfile

Uncomment these two lines

gem "broach", :git => "git://github.com/Manfred/broach.git"
gem "nap", :git => "git://github.com/qrush/nap.git"

Now you need to run gem bundle again inside the integrity app to add the new gems for the campfire notifier

$ gem bundle --only default

apache config

I’m going to serve it with apache, so I need to create the virtual host definition and enable it.

sudo vi /etc/apache2/sites-available/integrity

<VirtualHost myintegrityserver.com:80> 
	ServerName myintegrityserver.com 
	CustomLog /var/log/apache2/integrity_access.log combined 
	DocumentRoot /path/to/integrity/public 
	<Directory /path/to/integrity/public> 
		AllowOverride all Options -MultiViews    
	</Directory>
</VirtualHost>

sudo a2ensite integrity
sudo /etc/init.d/apache2 restart

When you browse to your server you should now see the integrity application running, but obviously with no projects built yet, let’s build one.

Name

We use the same name as our git repository on github, e.g.
web_app

Repository Url

This is the clone URL to your github project with .git at the end removed(current issue).
e.g.
git@github.com:username/reponame

Branch to Track

master

Build Script

We currently use a bash script to perform some tasks prior to running the test suite. We were using a rake task, but ran into some issues with rack1.1 so we’re using this method now.
e.g.
/path/to/bashscript.sh && script/cucumber

sudo vi /path/to/bashscript

#!/bin/sh
cp /path/to/database.yml config/
RAILS_ENV=test rake db:migrate --trace
/usr/bin/gem bundle > /dev/null 2>&1

Tick the Campfire Notifications and you’ll see some extra configuration options. We’re going to use this to send the result of the test back to our campfire site so we can all see the test results, good or bad 🙂

Subdomain

This is you campfire subdomain, so if my campfire site was at matt.campfirenow.com it would be

matt

SSL

Tick it

Room Name

This is the name of the room itself, not the URL to it. e.g., if my room was called matts’s pictures 1.0, then you need to put in this box exactly that, matt’s pictures 1.0.

API Token

This is your API authentication token from Campfire. If you login to campfire and click on My info you’ll see it there

Notify on success

Tick it, it’s always good to know when the tests pass, in a vain attempt to balance out the failures 🙁

Update the project.

Configure Github

Firstly I had to add the contents of ~/.ssh/config/id_rsa.pub as a SSH Public Key on my github account.

Now we need to configure our Post Receive Hook on the Repo we’re testing, so click on the repository, then click on Admin. In there click on Service Hooks and then select Post receive URLS

Now add one with this format

http://myintegrityserver.com/github/TOKEN

this is defined in config.ru

c.github       "TOKEN"

Once you’ve updated you’re settings you can test that hook. What should happen is the payload should be delivered to your integrity server and the test process should start. It may not, more than likely won’t, so I’d go back to my integrity project and try manual builds from there until you get it to work.

Try out a manual build at http://myintegrityserver.com to see how it goes.

Couple of tips

1. Always have tail -f integrity.log open, it will give you some information about what’s happening.

2. Output the results of the different commands happening in your build so you can see what’s happening in detail. I found this really helpful.

e.g. in the Build Scripts section you could do this

/path/to/bashscript.sh && script/cucumber > /path/to/cucumber.log

you can then tail that log during a build to output any debugging you need

3. The bash scripts and whatever code you put into the Build Scripts section of the app will run in the new build folder. If you’re having problems running anything just cd into that new build and try running the commands there. NB, you must be the same user that apache is running as, which is the same user that owns the integrity files and folders and you’re apps files and folder.

Now when anyone pushes to our master branch of our application on github, our complete test suite runs and we all get notified of whether or not the tests passed in our campfire room. Now the fun never ends 🙂

MySQL Replication

The situation

We’ve got a web application that uses a mysql database as it’s backend. Some of the data held in that application needed to be used in another web application, but only read, never written to. So what we had to come up with is a method of using that data.

Options

1. Use the Rails ActiveResource class

Using this method we could read restful data from the remote system directly, and use the ruby objects we pull into our site to display the necessary data.
This is a relatively easy method of achieving our goal, requiring minimal amounts of coding on the remote application and we keep one source of data.

Some of the downsides to this method are if the remote system goes away(network failure, mysql crashes etc etc), then our new web app falls over. Also the remote system will always have to do most of the grunt work, running the mysql queries, creating the xml, etc etc. If it’s a busy remote system this may have a negative impact on how both systems run.

2. Use the Rails ActiveRecord class

The downsides to this problem are the same as the ActiveResource method, with a couple of additional problems. We’d then have access to all the database tables, and there are some tables that contain sensitive data, so that would require some additional work. Also, you can only define one database per application, so if we wanted to add any tables to our new web application, we’d have to add them to the database that powers the remote application as well. That could prove very difficult to manage.

3. XML

This is really a less efficient ActiveResource type solution with the same pro’s and con’s.

4. Master/Slave MySQL Replication

The only real downside for this method is that we’d never done it before. After a bit of testing we quickly found out that we could eliminate all the problems of the other methods.

We can synchronise only the tables we want, so we wouldn’t run into problems with sensitive data.
If the remote system goes away then our system will remain unaffected and the data changes will “catch up” once the remote comes back online.
There is no additional load on the remote system as the new system will query itself.
If we need to add any tables to our new application then that will have no effect on the remote application at all.

MySQL Replication

Replication follows a Master/Slave methodology, in our case the master is the application that is already in place, and the slave will be the new application.

I’ve left out how to create the user for replication(slave_user) and give it the necessary permissions, you can find out how to do that in one of the pages in the article above if you don’t know how to. Also don’t forget to set the bind-address to the IP of the slave on the master, and open up the firewall on the master on port 3306 for the IP of the slave.

First off we need to enable binary logging on the master, set the server-id, the database we want to replicate, (we’ll specify the tables we want in the slave), and some other variables to keep the system from getting out of hand.

MASTER
sudo vi /etc/mysql/my.cnf

Either add these lines to the [mysqldb] section or uncomment and edit them if they are already there.

server-id=1
log_bin=/var/log/mysql/mysql-bin.log
expire_logs_days=10
max_binlog_size=100M
binlog_do_db=database_name

restart the mysql server

sudo /etc/init.d/mysql restart

Now we need to tell the slave mysql server that it is the slave, and also which tables to replicate. You can also set some other management options here.

SLAVE
sudo vi /etc/mysql/my.cnf

Either add these lines to the [mysqldb] section or uncomment and edit them if they are already there.

server-id=2
master-connect-retry=60
replicate-do-table=database_name.table_name

restart the mysql server

sudo /etc/init.d/mysql restart

As we already have data in our master system we’re using the mysqldump method to get the current data(via Sequel Pro). In order for the data in the master to be the same as that in the slave initially, we need to stop any commits on tables, or LOCK the tables of the master database, prior to taking the data dump for the slave.

MASTER
mysql -uusername -ppassword
mysql>use database_name;
mysql>FLUSH TABLES WITH READ LOCK;

This will block all commits until you close that mysql session. Export the sequel dump from the master now and import it into the slave. I did this with Sequel Pro and the import/export tools available in that.

We now need to grab some details from the master that we’ll need later on when we tell the slave to start replicating.

mysql>SHOW MASTER STATUS;
———————-——————————-—————————+
| File | Position | Binlog_Do_DB | Binlog_Ignore_DB |
———————-——————————-—————————+
| mysql-bin.000004 | 20060 | database_name| |
———————-——————————-—————————+

Now back on the slave we need to use these variables and some others to setup the replication.

SLAVE
mysql -uusername -ppassword
mysql>STOP SLAVE; (check if the slave is already running or not)
mysql>CHANGE MASTER TO MASTER_HOST=‘IP of master’, MASTER_USER=‘slave_user’, MASTER_PASSWORD=‘slave_users password’, MASTER_LOG_FILE=‘mysql-bin.000004’, MASTER_LOG_POS=20060;
mysql>START SLAVE;
mysql>quit;

Go back to the master and close the mysql session you started earlier. This will release the lock on the tables and allow updates and commits on the master again.

MASTER
mysql>quit;

That’s it, any changes in database_name.table_name on the master will be replicated over to the slave. If the master goes away for any reason, the slave system will still function and will ‘catch up’ when the master comes back online.

What we’ve now got working is a central data storage point that pushes any changes in it’s own data out to a remote system as and when the data changes. It’s pretty easy to add more slaves if you need to for scalability.

If you had a lot of people/systems all working on the same dataset, and you wanted to make certain that they were all using the same data, then the remote systems that only read data could be the slaves, and any systems that need to write data could use the master. You can also set up master to master replication, so that if some remote systems needed to write data as well, then they could.