you only need main building 3 to build a barracks. and you need the troops asap for raiding. you should have them by now, no wonder you're growing so slowly! :P
get troops, even just a few of them you can raid with all the time and double your production amounts. use injates tool to find zero growth people with low pop to start, and make your way up to medium pop people as you go along.