After a bit of hacking I set up a little Ruby script that outputs exactly that using the magic of a pam_motd
banner, so that when you log in you can immediately see how your system is doing. Getting it going is pretty easy and I’ve found it quite helpful so far – if you’re interested in running it as well, here’s how!
noupdate
from /etc/pam.d/sshd
and /etc/pam.d/login
Both the /etc/pam.d/sshd
and /etc/pam.d/login
files contain lines like this:
1 2 3 4 5 |
|
You’ll want to comment out noupdate
in both files, which prevents pam_motd
from updating before login. Normally you don’t want the motd
updating before login, since that could be a bit time-consuming and you don’t want to slow down the login process. However, if we don’t update it before login then you’ll get stale data, which is rather pointless for a status banner. And happily my Ruby script is very light-weight and adds less than 50ms to a login. Which, sure, not great, but also, not bad.
So ensure you comment out noupdate
before moving on to step 2.
statuser.rb
script in /etc/update-motd.d/
The script I came up with is pretty lightweight, and I’m sure I’ll spend some time tightening it up, but for my purposes it serves well. I tried to avoid too many abstractions: though my first inclination was to put the memory and disk stats in objects, keeping them as arrays helps make the script minutely faster. Similarly I used oj for JSON parsing since it’s the fastest available JSON parser, though the amount of JSON Consul returns is so small I think any parser would’ve performed adequately.
So, with no further ado:
To speak about the code generally, at the top I define a bunch of constants that you might want to change (like I thought using cute unicode characters for statuses was sensible, but if you don’t have a unicode-compatible system you should probably change those). system_status
queries for a lot of miscellaneous statuses and converts them into words and colors that are easily digestible – similarly, monit_status
and consul_status
do the same for my monitoring services.
Helpers at the bottom format strings and tabularize the data into a more lovely format. I tried to keep the stuff you might want to change at the top, but if you really want to get into the nitty gritty of displaying, tweak the private methods at the bottom of the class.
SSH into your server and you should see your new banner displayed proudly!
If it isn’t working you might just see your system’s standard motd. If that’s the case, try running the statuser manually – hopefully you’ll get an error you can debug a little bit more.
I know the statuser isn’t the most elegant Ruby ever, but it does work, it is fast, and also it’s easily extensible: you could add a lot more useful information to it very simply. I tried to err slightly on the side of conservatism with the data I chose, since I didn’t want the login process to take too long… but I would love to hear suggestions for other interesting tidbits to put in that you might want to see every time you ssh in. And, of course, as I improve it I’ll keep the gist updated.
]]>Want to do it yourself? Then read on.
There are two parts to this setup: first, compiling the software and creating packages from it. Then, downloading it on client servers. For the purposes of this cookbook, I’ll be referring to the former as “creating,” and the latter as “installing.”
Creating is pretty easy. We’ll have one generic create
recipe that all the other recipes can include, and a provider that does most of the heavy lifting. First, let’s set up some cookbook attributes:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 |
|
I’ve pegged our package creation to fpm 1.3.3, which has worked extremely well for us. Remember to include your actual access key and secret access key to actually upload this to S3! Also note the cache directory location. We’ll be using that down below.
Now the base create recipe:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 |
|
Here we’re just installing a bunch of packages that will be necessary for compiling whatever software we choose. And finally, let’s create a version of Ruby:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 |
|
Again, we’re just installing package prerequisites, compiling Ruby and a few gems, and then using a provider called symondsandson_packages
to do the real work. The provider is quite special, so let’s dig into it in a little bit more detail.
Of course, the package provider does most of the heavy lifting. In addition to creating the packages, it also supports installing them. Before we get to it though, let’s implement a little bit of abstraction so we can DRY up the provider a bit:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 |
|
This shouldn’t exactly be surprising stuff, and we’re about to use it extensively!
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 |
|
There’s a lot of stuff going on here, so let’s take it piece by piece.
First, we set some instance variables for use throughout the provider. We use them almost immediately in the install
action with the excellent s3_file Chef provider, which allows us to easily download files from S3. Tragically we are forced to do some Ruby gymnastics to set instance variables for the providers, since instance variables do not enter blocks appropriately in Chef… but once the file is downloaded from S3, we use dkpg_package on it to install it. If both the files was downloaded and installed, then the install
action updated its resource.
The create
action is just as easy. We pass in a bunch of arguments that fpm expects: here we make extensive use of the fpm_tng provider, which wraps the installation and use of fpm. Finally, we manually upload the created package file using Fog directly.
Using this provider and the create
recipes above, you should be able to start up a vagrant or test-kitchen instance using a recipe like symondsandson_packages::create_ruby
and have it automatically compile and upload Ruby to an S3 bucket of your choosing.
Now that we have the provider out of the way, installing packages is simplicity itself!
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 |
|
This simple and robust system has allowed us to package up any kind of software in a repeatable, efficient manner and then deploy it to multiple servers quickly. It’s saved our clients tons of time wasted in compilation, and was pretty interesting and fun to code besides. Hopefully you’ll find it useful as well!
]]>At Symonds & Son, we spent a lot of time and energy integrating our tests on CircleCI into a continuous deployment process that begins with three layers of testing and ends with automated cookbook deployment to the chef servers we manage. I’ll discuss chef continuous deployment in a later article; here, I’ll cover how we got Foodcritic, ChefSpec, and Test Kitchen all running seamlessly in Circle.
My good friend and coworker, Chris LoPresto, contributed greatly to the engineering innovations discussed here.
CircleCI has a number of great integrations: here we’ll use their docker service to create docker instances and automatically converge them with Test Kitchen. Before we do so, we’ll run foodcritic and ChefSpec on them as well just to make sure everything works as we would expect.
You’ll need to actually set up tests and get them running before getting them into continuous integration! Happily the process of testing cookbooks is relatively fast and easy: all three software tools Symonds & Son uses have great documentation and tons of examples on the Internet. Here’s how we set them up.
Running Foodcritic is pretty simple. Simply add the foodcritic gem to your Gemfile
and execute a command like this:
1
|
|
~FC003
instructs Foodcritic not to use a rule that guards for chef-solo. We intentionally do not obey rule FC003 as we use chef-zero locally and chef-server remotely.
ChefSpec is slightly more complicated. Your best bet is to follow the excellent installation guide at the ChefSpec README, since we don’t really do any customization on top of that: our ChefSpec tests are rather traditional. Here’s a sample from our cookbook that updates aptitude:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 |
|
The most complicated of the chef testing suites, Test Kitchen performs actual convergence on a platform of your choice and then will run automated tests on the created instance. Test Kitchen is also the most essential of the suites, in my opinion – performing static analysis and unit tests are all well and good, but the only way to definitively determine if your cookbook works is to actually ensure servers converge and that their internal state is correct.
Most of our cookbooks use Docker containers to perform testing. This has downsides: docker containers will not allow you to modify important files in their /etc
directory. For example, you cannot change iptables rules in a docker container. Additionally Upstart does not work at all in docker-land. Despite drawbacks like this, docker has many advantages. It is extremely fast and well-supported across testing providers. In fact, our continuous integrator of choice, Circle, provides first-class support for docker containers on their test VMs – allowing you to run docker Test Kitchen convergences directly on Circle.
Happily, the .kitchen.yml
that supports this is rather standard:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 |
|
The only special section to note here is the driver configuration: we’re using docker and setting it to privileged mode, which enables it to more exactly match a cloud VM.
Just for completeness’ sake, here’s a simple test from the afore-mentioned aptitude cookbook testing for the proper version of Upstart:
1 2 3 4 5 6 7 |
|
Once you have your three testing suites set up, all that remains is to integrate them into Circle.
CircleCI is my favorite continuous integration tool. It has a great UI and is really simple to set up with GitHub: it also has great support for third-party plugins, and even better, lets you SSH into a failed instance to run tests yourself and figure out exactly what went wrong.
For our purposes, we need CircleCI to properly install chef and then run all the tests on our cookbook. As it turns out this is not all that difficult to set up: you’ll want a circle.yml
that looks a little like this…
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 |
|
There’s a lot going on here, so let’s dive in at the top!
First, the machine.services
directive informs CircleCI we want docker to start on our Circle test VMs. Without this, the kitchen specs won’t work at all.
Next comes the dependencies directive. Chef provides the excellent ChefDK download which bundles everything we care about: chef’s own command-line interface, Berkshelf (for managing cookbook dependencies), and all three testing tools we’re using. We’re installing the official Chef .deb
distribution of ChefDK onto the Circle machine to get around downloading each tool individually.
Next comes a bit of gem back-and-forth. Until very recently, the GitHub kitchen-docker main fork was not kept up-to-date: we manually install a version of kitchen-docker that actually works, and then reinstall a better version of chefspec.
Finally, we copy a stub knife.rb
to ~/.chef/knife.rb
. You’ll need to set this file up yourself, but it’s intentionally pretty small. If you’re downloading any cookbooks from a Berkshelf-API server, you’ll probably need to include a valid private key for that server either in the knife.rb
(bad) or include it as an environment variable in Circle (good). Here’s what the knife.rb
should look like:
1 2 3 4 5 |
|
Once all the setup is done, running the tests is by comparison quite simple! We perform a berks install
and then run each test command individually. If you have your tests working locally, this should get them working properly in Circle as well.
Of course, even if you are testing your cookbooks, you’re really only half-way to heaven. After testing comes deployment: and this can be a little difficult with chef’s complicated keying and validation structure. In my next post I’ll detail how we automatically release and deploy cookbooks to their destination chef servers… stay tuned!
]]>My consultancy, Symonds & Son, has had a successful year. In 2014 we grossed almost quadruple what we did in 2013 – which makes sense, since it was pretty me just me and some part-time contractors back then. While I very much hope we experience similar success in 2015, I’m not exactly projecting continued growth of that magnitude – I think it’s likely Symonds & Son will finish next year in a similar place to this one, given my clients’ burn rates and my excellent client retention numbers. But I’m getting ahead of myself… let’s start digging into the nitty gritty of my books.
Symonds & Son has three full-time employees: me, my mother, and my father. If you’re curious how that works I made an earlier post about working with your family. We paid twelve contractors in 2014 for work as diverse as Rails application development, devops chef help, postgresql optimizations, iOS app creation, web and app design, and lots of accounting. Almost all (more than 96%) of our 2014 expenses were payroll and contractor payments:
Operations is how I’ve bucketed AWS, DigitalOcean, CircleCI, RackSpace, CodeClimate and GitHub subscriptions. Software, by contrast, is one-time license payments (for Adobe Photoshop, for example), and hardware is all my visits to the nearest Apple store or authorizing hardware purchases for contractors/employees.
See that 0.2% marketing expense? That was a very modest Google AdWords campaign. The result of that campaign was a reasonably good client lead that we were unable to follow up on, due to my inability to staff up quickly enough to satisfy it. In general, figuring out how to properly market Symonds & Son has been my biggest challenge… but more on that below.
The 96.5% contractors & payroll number might initially appear surprising, but I think in a consultancy it makes sense. My entire business relies on the strength of my employees – I’m literally converting hours into software, when you get down to it, and my clients expect excellent quality and superior craftsmanship from my company. Also there’s a fair amount of stuff that’s just not in my core competency: though I manage my business as best I can, I do not do our taxes or bookkeeping, and I’m happy to pay a really excellent accountant to do it for me.
Ultimately consultancies are about people, and I’m proud to say that my expenses reflect the value of my contractors.
But that sounds a little trite. This post is about showing the numbers, so let me prove how much I value the skills of those I pay. Here’s a chart showing how my money was divided among contractors and payroll, by skill:
By necessity, Rails includes a fair amount of JavaScript/HTML/CSS stuff that it would be pretty difficult to break out.
Marketing here is a social media marketer who’s helping me with branding and strategy – a separate expense from directly paying for advertising, and one that has been more useful anyway, in my opinion. Perhaps most surprising from a personal perspective is costs spent on chef: those expenditures are internal, buying the time of some really excellent talent from the chef community to improve and build on my infrastructure cookbooks.
One of my goals for 2015 is to acquire new infrastructure clients and package my existing infrastructure services as a software product, which we’re tentatively calling Cornerstone. More on that below, as well.
We had ten clients during 2014 and an additional four client possibilities that we failed to land, giving my company a conversion rate of 71%. That’s really good, but as you’ll see it’s only part of the story. Here’s how most of those clients found Symonds & Son:
Nearly all of my clients find me through the very blog you’re reading right now. (Do you need Rails development, application scaling, or devops services? Email me, I’m really good!)
Of those leads, we did not convert two friend referrals, one blog post, and the only Adwords potentials. Overwhelmingly, clients find and stick with me because of my writing. I think the success of my blog as a marketing mechanic is rather self-explanatory: if someone sees something I’ve written about that they really need help with, they’re very likely to pursue me and almost usher themselves through my sales pipeline.
This has downsides though – it’s not really scalable. Increasing the reach of my blog is difficult since it’s hard to properly advertise a (mostly) technical blog. In 2015 I hope to find more clients in need of devops and development services, particularly those I might not ordinarily reach with an article on consul or elasticsearch: say, a Vice President of Engineering or a Director of Technology who knows in general they need help, but not that they need cloud-scale chef deployments or expert Rails consulting.
As part of this shift I’ve spent some time redesigning and rebranding my own corporate website, trying to productize my core offerings as packages of services, rather than just services. Hopefully product clarity will allow Symonds & Son to sell more of what we do best in the coming year. I’m looking to roll out this redesign by the end of February, so that will give me 10 months to test it.
I’m proud to say that of my 2014 clients, all except two will be continuing into 2015 with me, giving Symonds & Son an 80% yearly retention rate. The ones who I’ve parted ways with I hope to see again; both are startups with funding issues. Considering the sample size is a bit small this doesn’t necessarily mean anything, but I like to think that our clients appreciate the services we continue to provide to them.
Speaking of services, here’s a rather interesting breakdown of Symonds & Son revenue by technology service:
Devops is always sort of a loose term – here I mean recurring and one-time revenue from customers who are enjoying our really excellent chef cookbooks and infrastructure administration, and those who paid us to set up cloud installations for them. This includes postgresql and elasticsearch design and development by experts, for example, if you’re correlating this to the expenses by skill chart from earlier.
And this revenue breakdown rather closely mirrors that chart, which after all makes sense: I’m trying to match customers’ needs to the skills of my organizations, and if I were paying a ton for, say, elasticsearch but didn’t have many customers utilizing my contractors with those skills, I’d have some expensive people sitting around doing nothing… far from optimal!
Though I’m really happy with the iOS apps Symonds & Son developed in 2014, they unfortunately produce the least revenue of any of our services. And I’m not sure I want to change that much: in 2015, what I’d like to see is increased revenue from devops. I’ve very much enjoyed the operations contracts we’ve had, and we’ve invested a ton in our own cookbooks and development processes around those cookbooks – a whole suite of software and processes that we’re calling Cornerstone. Using Cornerstone more would be a tremendous victory (and would be super fun, besides).
2014 was a really great year for Symonds & Son. But that sets up a certain expectation of more success and more growth, when in truth it’s impossible to predict what the coming year will bring. I personally view our present clients as extremely stable and loyal, so hopefully there’s nowhere to go but up – but overall I hope to chart a conservative course for my company, erring on the side of growing properly rather than overexpanding.
And that’s what the internals of a small consultancy looks like! Hope you’ve found this interesting – and if you’re looking for Rails development or devops help, don’t be afraid to drop us a line. I think you’ll be very pleased with our services.
]]>This setup assumes you have your servers connect to chef for both provisioning and deployment; additionally, that your servers are named at least partially after the applications they serve. (I find this to be good practice anyway – as cute as it is to have server names themed from Teenage Mutant Ninja Turtles, in practice you just end up getting confused about what they do.) All the servers should be able to communicate together on a secure network. That’s what you’ll run consul on!
First, you need to install consul on your servers and get them communicating together properly. As you can tell from my links, the consul documentation is extremely well-equipped to get you a fully functional consul cluster: just follow it and you’ll be fine. Once it’s all installed, you’ll want to drop a watch like this in your app server’s consul config directory:
1 2 3 4 5 |
|
Note the sudo above: chef-client must be run as root, but this shouldn’t be too big of a concern. Just make sure that your consul service is running as a sudoer that can execute only that one command and you’ll be fine.
Getting CircleCI set up properly with your GitHub repository is covered in great detail at CircleCI’s documentation. Make sure your tests pass on CircleCI before you continue.
In order for this to work, CircleCI will have to be able to propagate that event to all your servers. The easiest way for it to do this is to have it SSH into one server and run the consul event
command. From there, consul takes over to make sure all the appropriate servers receive the message. So let’s create an SSH key for our CircleCI user.
1
|
|
Add the resulting private key to CircleCI and create a corresponding user on one of the central servers of your cluster. In my setup, I have three separate instances all running consul in server mode: I assigned one of them the DNS consul.example.com
and created a user there to accept Circle’s SSH key. Notably this user does not need any sort of permissions at all, so leave them off the sudoers.
Now you need to change your circle.yml in your project, so that after a build CircleCI will initiate the deploy. This is pretty simple:
1 2 3 4 5 |
|
Note the <app_name>
filter there for nodes. I find it a good practice to name a node after the application running on it; so I might have one called app1-web
and another app2-worker
. This is helpful for performing consul node filtering: by providing -node <app_name>
, you’re ensuring that the consul event is only propagated to servers that actually run the targeted application.
The next time you push a commit, CircleCI will automatically SSH into your server and execute that event. Your application servers with the watches on them will run chef-client
and pull down the latest code, automatically deploying the most recent version of your application.
This is obviously a fairly simple application of consul, but I found it extremely easy to setup and drop into my existing application provisioning and deployment process. consul provides a lot of automation and power, however, and you’ll find it perfect for helping to bridge the small gaps in your provisioning system.
]]>This was a really dense talk, as one would expect about a completely new implementation of Ruby. Rubinius aims to be a performant, compliant Ruby – Rubinius X means version 10 of Rubinius (8 versions ahead of MRI!), and has some very ambitious goals: perhaps the most interesting of which is ensuring the future of Ruby.
It’s no secret that Ruby is old now. The question is, how do we keep it relevant even in its increasing age? Brian’s hedge against winding up in a programmer nursing home is Rubinius X, which has a lot of interesting features:
The greatest trick the devil ever pulled was to convince language designers that strings are arrays of characters. - Brian Shirai
The claim is that Rubinius (and Rubinius X) are usable today, immediately, in your MRI app with only minor changes to your Gemfile.
As exciting as all this is, I am personally a little skeptical. Rubinius X promises the moon and stars but we saw little actual code demoed, so I’m not sure how many of these features are actually available or how best to use them. I would be interested in performance comparisons to MRI and seeing an example of how this would improve my existing code and simplify my app’s design patterns before I’m totally sold.
By comparison to the previous, this session was quite light. Extremely helpful for the Rubyist not yet familiar with resque or redis, Recommendation Engines with Redis and Ruby focused on a specific case study for implementing a recommendation engine in redis.
While the talk was quite fascinating, I was personally hopeful it would focus on the statistical methods of recommendation (using something like k-clustering), which it did not. Instead, the case study discussed implementing a solid and performant recommendation engine taking advantage of redis’ native features: inverted indexes, clever key TTL tricks, sorted sets, and queues, queues, queues!
resque workers performed the actual recommendation work, which was based purely on the relative apparent interest of users in tags, and bubbling up posts and users based on the users’ own tagging behavior.
Though an interesting and conversational case study, and probably very informative for a beginning Rails user, this session did not really discuss anything revolutionary – probably because the sample app seemed to be a few years old at this point. resque is old news now, and redis even older, but the design patterns discussed were still universal and helpful.
Devise is the most popular Rails authentication framework, and this was a very useful presentation on what Devise actually does and how it does it. Significantly more interesting than just a read-through of the Devise README, this presentation was simply a walkthrough of the gem and how to customize it.
The foundation of Devise is Warden. If Rack is HTTP infrastructure or ActiveModel is ORM infrastructure, Warden is authentication infrastructure. Warden is Rails middleware that provides user session management, failure handling, and lifecycle callbacks. Devise integrates Warden’s magic seamlessly with Rails, providing model, router, and controller/view patterns to handle almost all common authentication cases.
Of course, Device’s Rails opt-in model integration is great. It provides modules such as session expiration, IP tracking, and account confirmation all in your user model, if you want it. And on top of that you have routing helpers and tons of default controller actions and views. Additionally, Devise is sensibly secure by default – after all, you don’t want to be concerned with implementing security! Security experts should be taking care of that. So Devise uses bcrypt, encrypted tokens, and prevents timing attacks by default.
Devise is a set of reusable, extensible shortcuts for getting authentication cleanly and seamlessly into a Rails app, and honestly everyone should be using it in their app if they aren’t already. This session was a concise overview of how best to implement it and configure it.
Initially I was concerned this would be a session about visual design, but actually this was a very cogent presentation on code design, refactoring, and the proper way to structure code. That said, it started with a rather controversial question: who likes TDD? Who doesn’t like it? Hands were raised and responses mixed, with most people falling somewhere in between.
Though doing upfront design has a reputation for not being particularly agile, Let’s Do Some Upfront Design argued quite clearly that planning before starting a project can reap immense rewards down the road. Particularly using sequence diagrams and separating the coordinators (the what) from the processors (the how) can simplify your thinking about a system and lead to better abstractions and inheritance hierarchies.
While this was a really helpful talk from a design perspective, the focus on testing quickly became very abstract, disappearing into the realm of mock objects far too rapidly for my tastes. (What exactly is the value of tests if they’re all testing mocks?) Still, for illustrative purposes the tests were very well-written, and seeing the value in both clear tests and clear models went a long way to showing the value of early planning.
The overall value that the upfront design achieved is undeniable, and the talk itself quite clearly illustrated that with many examples and clearly argued points.
Of all the sessions on the first day, this was the one I was looking forward to most! My consulting business, Symonds & Son, has been quite successful, but success brings with it the reality of contract negotiations and the potential for legal difficulties, areas I am not yet too familiar with.
Though taking only half an hour, Protect Your Code summarized the main legal issues facing developers: copyright law, licensing, open source software, work for hire, licensing and assignment, trademarks, and contract law.
Perhaps the biggest take-away for me was understanding code ownership a little bit better. If you are an employee of an organization and you develop code in your day job, the company owns that code. You don’t have to sign a piece of paper for this – it’s just that way by default. As a private contractor, everything you create is yours – except work for hire, where you must assign copyright to the person hiring you.Either as an employee, employer, contractee or contractor it’s vitally important to understand the ownership of created code and how contracts change that ownership.
And that led naturally into a discussion of licensing with open source software, particularly the tainting nature of the Gnu Public License. When used for hobbyist projects it’s quite acceptable, but the fact that everything it touches becomes open-source can be quite a nuisance for enterprise-grade software.
While fascinating through and through, this presentation make me concerned to review the contracts I use in my business. Probably this is a sensible warning to all consultants: have a good lawyer look over your agreements. Hopefully it’ll be wasted money, but if it’s not, it’ll have been a very worthwhile exercise.
Domain Driven Rails started with a boxing match between DHH and Uncle Bob.
Throughout the first day as a whole, there was definite tension between the simple, CRUD-based applications DHH advocates and the complicated, enterprise logic of Uncle Bob’s hexagonal architecture. (To anyone who’s been to Ruby conferences before this will probably come as no surprise.) Of course there’s no resolution to be found between the two, and happily, this talk took a very sensible middle-ground.
It also included lots of actual code and many interesting statistics from Code Climate on a real, actual project: reverb, a guitar marketplace. Though the reverb app is monolithic by any definition of the word, with a thousand classes and a hundred models, it still manages to have low churn, high code quality, and quick development iterations. Indeed, they defended their huge application with a very sensible quote from Martin Fowler about the trend towards separating apps into distinct services:
While small microservices are certainly simpler to reason about, I worry that this pushes complexity into the interconnections between services, where it’s less explicit and thus harder to figure out when it goes wrong. - Martin Fowler
reverb’s refactoring secret? Plain old Ruby objects! The presentation argued quite convincingly in favor of rejecting “skinny controller, fat model” for “skinny controller, skinny model, and skinny domain objects sitting between the two.” This domain layer encapsulates specific kinds of behavior – say, a ReturnOrderMailer or a ProcessUserCreditCard – that can decorate models but are not necessarily part of them.
This definitely takes the better parts of DHH’s philosophy of simple POROs while rejecting MVC as the be-all, end-all of Rails application design. Decorators, combined with events and event listeners, helped to significantly lower the architectural complexity of reverb without having to break it out into many disparate parts.
Overall a very solid argument, though I’m still not completely on-board with the idea of hundreds of classes each controlling a minute part of a model’s behavior. It clearly works for reverb, though.
Providing a straightforward, sensible introduction to Go, Go for Rubyists gave us a glance at Go, complete with code samples and helpful walkthroughs. Once again, the keywords for this presentation were “performance” and “concurrency,” which should be no surprise to anyone who’s even slightly familiar with Go.
And it’s hard not to be seduced by the allure of Go. It’s pretty great – fast compilation, statically linked, highly opinionated, lots of tooling, and all that while still human readable.
Concurrency is a first-class citizen in Go, based on routines and channels. A routine is sort of like a very, very lightweight thread. Routines communicate not by sharing memory, but by transferring data over channels. A very clear separation that makes it easy to reason about multi-threaded applications.
It’s always interesting to see other perspectives and different languages. This was a well-done presentation filled with demos and language samples, so it was clear and obvious what the benefits of Go would be for a Rubyist. If you also find Go interesting, you should go to golang.org and check it out yourself.
A fascinating discussion about using software in the public sector, Resolved dealt mainly with the experience of programming for large government entities. Spoiler alert: it sounds pretty difficult.
Though governments make use of many of the same Rails tools any of us would find familiar – Heroku, postgres, jQuery – the talk dealt a lot with the many different data formats that bureaucracies seem to inevitably accrue. If you’re lucky, your city might have a bunch of CSVs on-hand. But if you’re not, you’ll have a lot of scanned documents you’ll likely need to parse for data.
The presentation had some code samples on how to do this via OCR with Google’s Tesseract, falling back on Mechanical Turk in the case of ambiguity. Overall a clever solution that would probably prove extremely helpful for the (likely many) governments with legacy scanned data.
It’s rare to hear about the use of Rails (or just software development in general) outside of the isolated bubbles of startups and the larger but still isolated bubbles of large corporations. Resolved had a lot of great insight into this underreported facet of Ruby usage.
The Functional Web argued that Rails must evolve to meet the needs of a changing Internet: web sockets, streaming responses, and event-based connections all cry out for a Rails solution that ActionController and Rack cannot easily provide. While it’s easy to agree that these are all things Rails is terrible at, I’m not sure if Rails will ever be a great tool for solving these sorts of problems.
Ultimately Rails is intended to serve web applications, and thus must return a response within a reasonable period of time – as the presentation pointed out, unicorn and delayed_job (or preferentially sidekiq) can get you pretty far for both synchronous responses and asynchronous workers.
Once again, that bogeyman of Rails, concurrency, was evoked. But Rails is actually quite performant even in high-load situations with the proper application optimizations and server configurations. Concurrency issues, in my experience, primarily occur if you’re doing something that should not really be done in the request lifecycle – like serving huge files or processing tons of data.
And as if to illustrate that the presentation started discussing the best way to spawn a thread in a Rails controller to do both those things. This is a pretty significant anti-pattern for the reasons the talk itself described; it’s semantically unpleasant, incredibly slow, and generally just the wrong place to do this.
Criticism aside, the Functional Web sensibly pointed out that Ruby needs better thread implementation and support. But if Rails is ever going to be a sensible server for web sockets and streaming responses, it’ll have to look very, very different than it does today… and I’m not sure that’s such a good thing, when excellent tools to accomplish these goals exist already.
And that’s all I have for day one of the conference! Tune in tomorrow though, there are a lot more presentations and I’m quite excited about a few of them.
]]>Maybe it’s because President Obama told you to learn computer science, and you’re not gonna argue with the President.
Or maybe it’s because you hunger to make something – a product other people use – and you’re tired of a job where, at the end of the day, you wonder what exactly you did or why it even mattered. And hey, making software is easier and cleaner than making furniture.
But realistically it’s probably because of the money. Glassdoor claims that software engineers make a median salary of $85,000, which is nothing to sneeze at. Even the lowliest programmers can look forward to taking home about $60k, and if you have a talent and drive for programming, then the only limit to your salary is your ambition.
All well and good. But how do you go from a casual smartphone owner and recreational computer user to actually making software? And then how do you turn making software into real money that appears in your wallet and bank account?
After my article on consulting in Rails, easily the most-asked question I received was: “How do I become a programmer, and what do I do when I get there?” No one is born with a keyboard strapped to their hands: every programmer you know learned how to do it. I did too. Here I’ll describe what I found most helpful for getting into programming, how I’d do it again if I had to, and what you should keep in mind if you decide to set off on the programmer’s path.
This post is extremely long. But hopefully by the end of it, you’ll have some clear ideas and inspirations for how to turn yourself into a real, honest-to-goodness programmer.
To become a programmer, you don’t need a computer science degree, hipster glasses, youth, or even beauty. (You also don’t need a particularly strong math background, in case you’re concerned.) That said, there are some things you really do need – if you can’t put a check-mark next to these prerequisites, you might want to seriously reconsider a journey into the realms of computer science.
A love for computers. If you can’t see yourself working in front of a computer 8+ hours a day, happy as a clam, then I appreciate you reading this far, but this career path probably isn’t for you.
Sacrifice. Most software, tools, and books to learn computer programming are free. Some of the best ones, unfortunately, aren’t. You won’t have to spend thousands of dollars (unless you need to buy a computer), but it’s probably reasonable to expect an outlay of a couple hundred to purchase books and software. And that doesn’t even factor your time into it – you have to put in the work to get out the results, so expect late nights and early mornings.
Drive. The going will likely get tough – really tough. When doing something outside of your comfort zone, it’s easy to get disheartened and lose your way. How long you can stick it out and keep coming back will determine whether you end up with programming as a hobby, a mediocre skill, or a mastery. Unfortunately there’s no shortcuts here. You just have to do it.
Still with me? Good.
Before you even set off on this journey, we should consider the end goal. Specifically, what do you need to know in order for a potential employer to consider you a viable candidate for a programming job?
This prospective employer will look for three things when evaluating you for a programming job:
Actual knowledge of the programming language. Your employer will vet your technical competency by examining your existing code and asking you specific technical questions: for Ruby, the classic chestnut is “what’s the difference between a block, a proc, and a lambda?” Though you’ll get bonus points for knowing everything 100%, no one expects you to have your language memorized offhand or even know the minute details of its operation. They’re looking for breadth of knowledge, where you’d go to look stuff up if you didn’t know it, and a general feeling you’ve been around the block at least once or twice. In your code samples, they’ll want to see competence and fluency in the language in question.
Having used programming in a professional capacity. Even if you know your language, you’re no good to anyone if you can’t apply your skills and complete projects. An empty resume with a note at the top that you love to code won’t get you hired anywhere. The unfortunate reality of the labor market is that everyone only wants to hire proven winners. You have to establish you are one if you want to be employable. That said, you needn’t have created enterprise-level software as part of a twenty-person team – personal projects hosted on GitHub or Bitbucket are more than sufficient to get hired.
Demonstrated ability to work with other human begins. Yes, you want to be a computer programmer, not a diplomat, and yes, your job is to code: but you’ll probably end up talking a lot to your boss, coworkers, and possibly even customers. You have to be able to put people at ease and make them like and trust you, or else your career won’t go anywhere at all – and you certainly won’t make it past even a casual interview.
Well, I like Ruby. It’s beautiful, functional, easy for beginners and powerful for experts. Professionally you’ll use it either to create web and mobile applications (with Ruby on Rails or RubyMotion, respectively). Python is also great for many of the reasons that Ruby is, and generally more popular for non-web purposes, so you can’t go wrong with either.
Lots of people swear by Java, and if you’re in a big company it’s probably much easier for you to get buy-in for a language backed by Oracle than one created by a bunch of people on the Internet.
If you want to create graphically intensive computer games, you’ll want to brush up your math a little bit and check out the grand-daddy of programming languages, C (and its offshoots and derivatives, C++, Objective-C, and C#, depending on your platform). C is hard to start with, though, and has a lot of complicated concepts – if you’re just beginning, a language slightly friendlier to humans would smooth the learning curve.
It’s hard to go wrong with any language, though, since most of the skills you’ll learn in one will be more or less portable to another.
With no further ado, a step-by-step list to go from zero to hero! If some steps are too remedial for you, just skip ahead until you get to a point where you think it’s reasonable to follow along.
Yes, you might use one every day and carry one around with you in your pocket. But what do you really know about these mysterious machines that make your life easier? Probably all too little!
Even the computer languages that look very much like English require you to know something about the interior operations of a computer: you don’t have to be an expert, but a generalized idea of how things work will be necessary for you to get into programming. For example, if you don’t know what files are, what memory is, or how to find your IP address, you should start doing a little research. I suggest Computers for Beginners, which can be dry at times but covers a lot of ground very efficiently.
It would probably also be helpful to learn a little bit about the structure of the Internet if you intend to do web programming of any sort… and these days, almost everything seems to involve a little bit of web programming. HowStuffWorks has a good explanation of the general structure of the Internet that isn’t too technical.
Programming languages can be very different indeed; but at their heart, they’re just instructions for telling a computer what to do. Since computers are quite similar, most programming languages rely on many of the same concepts.
This is where programmers start spouting jargon like recursion, iteration, and conditionals. Though those are relatively big words, the concepts underlying them are quite simple and learnable. There are a lot of great places to start learning the basics, but I have a soft spot for code.org’s basics lessons. It might seem more like a game than programming… and that’s a good thing.
Or, if you’d prefer disconnecting from the computer for awhile, I would recommend taking a look at the fabulously fun RoboRally. It gives players an idea of how computers actually process instructions – and how instructions that might initially seem perfectly reasonable can lead to horrible results.
Now that you have a solid foundation, it’s time to start learning your programming language!
There are likely a lot of extremely good resources out there already to help you. I learned from some of the classic books of Ruby programming: Agile Web Development with Rails and Programming Ruby.
Remember to consult the documentation! Almost every modern programming language has extremely extensive documentation, and probably a dedicated online following that love making those docs easier to read and understand. A little bit of Googling on even the silliest questions will probably yield a treasure trove of information… some useful, some not. (Try, for example, “why is Ruby named Ruby?”.)
And while you’re learning programming…
It’s never too early to apply your skills, even if you feel you have no skills at all! Having a project in mind is great and will help direct and guide your learning. There’s no such thing as “too ambitious” when it comes to a personal project – want to create the next Facebook, manually process credit cards, or keep track of millions of items in a warehouse? Well guess what, computers are great at almost every task imaginable and you’re learning how to apply their incredible power to the problems of your choice. Go to it!
Yes, your code will be completely terrible. Yes, you’ll probably grow bored or frustrated with the project and abandon it. Yes, the mere thought of these first steps will cause you great embarrassment in the future (or possibly present).
But using your code in a project will reveal aspects of your chosen language you’d probably never discover through a book or a tutorial. This is the sort of battle-hardening that you only really get on the field: you have to be in the trenches to know this stuff. For a Ruby example, you might have to figure out asynchronous jobs or how to create QR codes.
Again, Google is your best friend here. One of the greatest triumphs and most unfortunate realities about programming is that someone else has already done what you’re trying to do. You can learn quite a lot from their experiences and whatever code they’ve created. Don’t steal, that’s incredibly unethical. But feel free to fork!
Finally, always keep in mind your goals. You need to prove that you know your language, you have to have a track-record of using it in the real world, and you have to show you can work reasonably and responsibly with people. This is where you start building your resume for your future career, so keep in mind your objectives and work towards them.
You might find yourself doing this as you get even slightly proficient: the adage that when you have a hammer, everything looks like a nail is very true. But that’s not necessarily a bad thing!
When you create a program for yourself, don’t hoard it. Release it on GitHub and include excellent documentation and concise tests. This is true even for what you’d consider really minor pieces of code – it’d probably help someone, and having open-source code allows potential employers (and potential clients) to see what you do, what your code looks like, and your process for creating it. More than anything else, GitHub has become my go-to tool for evaluating potential candidates for a position.
I don’t mean to scare you away if you intend only to work on closed software, but having code out in the wild lets people get to know you and how you work. It speaks directly to the three objectives I mentioned at the beginning of this piece. You need to establish your bona fides, and using programming personally – and releasing the result into the world – will prove you know what you’re talking about
At this point your code still probably isn’t anywhere near professional-caliber. Guess what? That’s fine! As long as you can still use it to do things – even if it takes you awhile, or the result is unpleasant, or there are a lot of “gotchas” for your poor users – you’re already a programmer. If you have a job, start trying to work coding into it.
When I was in college I had an internship where I was expected to go over huge CSV files by hand. It was a ridiculously unpleasant job… but a small amount of Ruby and a really ugly regular expression saved me an enormous amount of manual parsing and let me drill down to the problem spots immediately. When I gave the company my program, they in turn gave me a really stellar recommendation that started my career.
Unfortunately, right out of college, all that recommendation landed me was a job doing tech support, but I managed to use my bad programming abilities there, too! The company I was doing support for had a terrible process for dealing with data files – really big ones, gigabytes large. We’d go on-site, burn the files we needed to a DVD, take them back to the office, analyze them there to find out what was wrong, load the fixed files back onto another DVD, then go all the way back to the client.
I wasn’t a great programmer: in fact, I wouldn’t even say I was a particularly good one. But I made a little Rails site that accepted enormous file uploads, so we could just upload the file and someone at the home office would download it. It crashed all the time, it took an eternity to upload, and it had the jankiest login system you’d ever seen. But it saved me and all my coworkers a trip back and forth to the office… and it looked great on my resume when I decided to switch jobs.
As your skills wax, you’ll likely want to start finding real, professional opportunities to exercise them. It might be time to start interviewing for a new job – or, at the very least, changing your existing role to officially incorporate more of your hard-won programming abilities.
Once you get to this point, the only way to improve more is to shift programming from a side-project to your full-time gig. Your resume will likely be fairly scant for an engineering position. This is okay. You aren’t looking to be a senior systems architect (yet): most likely you’ll get something at the bottom of the coding totem pole. But this is your first step into the real world of employable programming skills! Be proud that you’ve made it this far.
At this point in my career I worked at a very small Rails consultancy in Chicago’s River North neighborhood. The pay was low but the coworkers were extremely friendly, and some were even minor Rails celebrities. It was a great lily pad for a year.
Take the opportunity to learn from your coworkers. Many of them probably won’t be much better than you, which is also okay. Learn from their mistakes just as you learn from yours and always strive to better yourself. Keep reading books on programming, both your chosen language and general practice. For example, Smalltalk Best Practice Patterns is an invaluable programming resource even though you’re likely to never use Smalltalk.
And always keep in mind your goals: knowledge of your language, proof of having used it professionally, and demonstrated ability to work in a team. Consider how your work will look on your resume or what ammunition it will give you during an interview. And aalways be on the look-out for excellent opportunities for yourself. You’re the only one looking out for your own professional development, so take it seriously.
One of the unfortunate realities of the job market is you’re much more likely to get a raise and better responsibilities by switching jobs. (It’s depressing but true.) Once you’ve put in your time as a beginner programmer, it’s time to spread your wings and fly to a company that has more challenges, better opportunities, and hopefully higher pay.
Briefly, I’d like to mention that you’ll probably experience some imposter syndrome as you interview around. Your new skills are likely to feel fairly unfinished, so doubting yourself as you look around – and wondering whether or not you can accomplish what prospective employers ask of you – is natural.
Here’s a little life pro tip: almost everyone suffers from the belief that they aren’t as competent as they are. A good attitude, great ability to learn, and accomplished programming skills will get you much further than you think. Don’t be afraid to take a chance on yourself. The likelihood you’ll fail is very tiny, and the probability you’ll succeed and learn a lot makes such a small risk worth it.
At this point this list is likely to become less useful for you, since, congratulations! You’re pretty well a programmer. But, if you’d like one piece of career advice from one programmer to another: continue to seek new and exciting challenges. Never rest on your laurels.
I mean this in a very practical way. As I mentioned, switching jobs is a great way to get a raise, but also it’ll give you new problems to tackle. Never stop building your resume and you’ll always have job security. More than that, continuing to grow your programming skills will provide a solid foundation for whatever you want to do with your life and provide intellectual stimulation for years to come.
And from here, your path is yours to follow. I’d love to know how it goes though, so drop me a line as you progress!
]]>This is all bunk.
If you’re an engineer, then your job is to code. Your job is to be good at it. No, more than that: your job is to be great at it. Anything less is a disservice to the people who’ve hired you and a disservice to yourself. You should work constantly to succeed at your job. Working towards anything else is a distraction.
I find the point of view that creating code is the least of our concerns particularly troubling, since the skill our profession provides is so unique and valuable. Usually you don’t see agile coaches, salespeople, or business development professionals jumping in and coding. And when they do, they’re not great at it. Why should they be? Their talents are of a different sort. So why do we constantly receive exhortations to do their jobs? Does having the ability to code make us great at marketing? Or leading a team? Or being a chef?
I’ll tell you what coding does make us great at: creating computer programs. I mean this unironically. Our applied skills produce amazing products, save hundred of hours of human labor, and make people laugh or cry. The knowledge that enables this is, for most of us, hard won. Retaining our engineering position is a lifelong process of learning, changing, and growing. It’s the Red Queen’s Race to keep up with our peers and our changing world. The fastest we can run will result in us merely standing still… and if you don’t run as fast as you can, you’ll be left behind.
Of course we must communicate what we’re doing to our bosses, our clients, and our process managers. But this doesn’t make you a boss, a client, or a process manager. If they don’t understand what you’re doing, despite your communication, then that doesn’t make you bad at your job. It makes them bad at theirs. This is a hard spot to be in, and while the solution might be to try to do their job for them, that doesn’t change what your job is. (To reiterate: your job is to code.) If you’ve taken their responsibilities on top of yours, that doesn’t make you great. It means something has gone very wrong indeed.
Coding is a form of communication. As you become better at coding, hopefully your communication skills will follow – and this allows you to better interact with others. And while there’s nothing wrong with using new skills, don’t become distracted by them. You have a talent, a knowledge, a skill that others in your organization lack. Your job is to employ that skill as efficaciously, as completely as possible. And that should be taking up 99% of your time in any given day.
Leave the strategy to the generals and the garbage collecting to the janitors. Do what you do best, do what you love. That way lies success, self-actualization, and happiness. Become distracted and soon you will be a general or a janitor: there’s nothing wrong with that, but their jobs aren’t coding. And if you were hired to code, if you want to code, then your job is to code. It’s as simple as that.
]]>You should get it. And when you do, use these configuration settings to make your experience even better.
You’ll want to enable a bunch of modules in your .zpreztorc
, which is the configuration file for everything prezto. Here’s my modules:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 |
|
They’re not kidding when they say the order matters: prompt
must come last, history-substring-search
must come before it, and syntax-highlighting
must come before that. The result of messing this up will likely be a ton of Zsh errors output to your shell after every command.
Some highlights from this list:
history
and history-substring-search
work together to bring you one of the most awesome features of zprezto: when you type in a command and press up and down, you’ll only get history results of you typing that command. Super helpful for typing in ssh, then going back in time to see your previous ssh servers.
syntax-highlighting
gives you some fish-style highlights. Executable commands will turn blue, incomplete or unrecognized commands will turn red, folders get underlined… it provides immediate and visceral visual feedback to what you’re entering.
directory
gives you some awesome shortcuts for the directory stack: d
to see all the directories you’ve entered, then the numbers 1
to 9
to jump to a previous directory you’ve visited.
And that’s just a start. Visit the prezto modules list and view each individual module’s README to learn more about them.
After modules are done, get your identities into the SSH agent! There’s a configuration block for that later:
1 2 |
|
I also set up a custom theme for myself, the source of which I’ll include below. If you want to enable that theme (or a different one), you’ll also need to specify it in your .zpreztorc
:
1 2 3 4 |
|
You can find the full text of my .zpreztorc
as a gist.
.zshrc
is loaded in every interactive shell: use it to set up your Ruby switcher, important aliases, and anything else you need for maximum awesomeness. Mine is fairly self-explanatory:
There are a lot of cool themes out there. But I was pretty used to my old Zsh theme, which was simple, clean, and had an obvious indicator when your last command failed. For an example of what it looks like, see the screenshot at the start of this post.
The arrow on the left shows green when your last exit code was 0, and red when it was 1. It includes the branch you’re on, with an indicator of whether you have uncommitted files on the right-hand side. It does a ton more as well, and here’s the source for your use:
Place this in ~/.zprezto/modules/prompt/functions/prompt_josh_setup
and then select it in your .zpretorc
for best results.
prezto is easy to configure and presents a lot of powerful options: you’ll want to spend some time getting used to it. But it’s also really enjoyable. Once you start learning the power behind your shell and how easy it becomes to access, I guarantee that you’ll never look back.
]]>How magical? Well, imagine super-fast command line deploys, seamless cookbook updates, great chatbot and application integration, then marry all those things to AWS autoscaling via elastic load balancing. One use case for my client: TravisCI automatically creating servers, running remote acceptance tests on them, then destroying them afterwards – all while notifying chatrooms of its progress. Now that’s assurance your code will work in production! Really, the sky’s the limit here for awesome integrations.
I’ve learned a lot in the process of implementing this setup. If you’re looking to give OpsWorks a go for your next project, here’s some hints and tips to make get started on the right path.
You’ll want to test all of your OpsWorks recipes locally – how else can you be sure they’ll work remotely? Vagrant is the ideal tool for making this happen. You’ll want to download the same AMI and the same version of chef that OpsWorks is using: ubuntu 12.04 and chef 11.10 respectively, for me. Here’s how to do that in your Vagrantfile
:
1 2 3 4 5 6 7 8 9 |
|
This requires the vagrant-omnibus
plugin, which you can install with vagrant plugin install vagrant-omnibus
.
OpsWorks expects all of your recipes to be in one git repository that it can download. This may be bad practice for many chefs, but since it’s required here and it’s the cookbook repository format that librarian-chef expects and supports, you’ll want to download and configure librarian-chef.
Just the default librarian-chef configuration works, with one exception: you’ll want to strip .git
directories from the checked-out sources to prevent OpsWorks from becoming confused. That’s relatively easy to set up:
1
|
|
I store the source of my cookbooks in one repository and use an orphan branch of that same repository for the actual cookbooks that are installed and managed by librarian. This is pretty easy to set up:
1 2 3 4 5 6 |
|
Then in your master branch, set up your cookbook branch as a submodule in a subdirectory that librarian-chef will install to:
1 2 3 4 5 |
|
I have a small Rakefile that allows me to run rake
to sync my changes directly to the cookbooks branch.
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 |
|
Keep in mind this setup isn’t ideal for collaboration: if I had a lot of people updating the cookbooks simultaneously, I would definitely set up separate repositories. But for smaller OpsWorks projects, this works perfectly well.
Though it’s clever that OpsWorks has their own recipe syntax they’d like you to use, my advice is: don’t. If you ever want to use your chef recipes somewhere else – or bring chef recipes from elsewhere to OpsWorks – you’ll thank yourself for just using the standard recipe format. So instead of this:
1 2 3 4 5 6 7 8 9 |
|
Use the plainer, simpler:
1 2 3 4 5 6 7 |
|
The former syntax won’t work properly on vagrant, just for starters, which is a great reason all by itself not to use it. You’ll want to control what recipes get applied where through custom layers rather than OpsWorks’ special syntax.
OpsWorks inserts a lot of their own recipes into your cookbooks, and you can’t disable this behavior, even if you’re using your own custom recipes. This can lead to naming collisions that can be frustrating to resolve. For a Rails stack, I had to manually remove the unicorn
and passenger-apache2
cookbooks that led to merge errors with the application_ruby
cookbook. Thankfully, removing cookbooks in OpsWorks is pretty easy: if you have a recipe named exactly the same as an OpsWorks one, yours will replace it.
You’ll want to create a cookbook named after the offending cookbook (for example, unicorn
) and replace every file in the OpsWorks cookbook with a blank one. You can find all the OpsWorks cookbook sources in their GitHub repository. So, to continue the unicorn example, you’d make a unicorn
directory, a recipes
subdirectory, and three files: default
, rails
, and stop
. The content of all these files should be something like this:
1
|
|
Obviously you should only do this if you’re definitely not using OpsWorks’ cookbooks.
Get rid of your data bags, encrypted data bags, configuration yaml files: everything. Embrace OpsWorks as your centralized chef server and the primary authority on the state and setup of your application. Data bags are arguably chef smell at this point anyway, and OpsWorks continues their inexorable slide towards obsolescence. You’ll want to set up everything you can with sensible attributes in your custom application recipes:
1 2 3 4 5 6 7 8 |
|
Then pass overrides in your stack JSON. Your stack JSON is where you’ll enumerate all the settings particular to your environment: though I’m not incredibly happy with this setup, as it’s not versioned, AWS makes it easy to copy stack and layer setups really easily, so in practice it’s not difficult to update multiple stacks or create a new one from sensible defaults.
So what’s the real advantage of doing this whole song and dance? Using the AWS API, you can command and control your servers (and all your attached AWS stuff) with an ease and simplicity you can’t achieve anywhere else. But for more details on that, you’ll just have to stay tuned for my next post, which will discuss all the awesome things you can start doing with OpsWorks once you have it set up properly.
]]>Don’t do any of this until you have a fully-provisioned server working exactly as you’d expect. Make sure auto healing is disabled for the layer. The rest of the steps assume you have such a properly set-up instance, a well configured layer, and that the original image was Ubuntu 12.04.
Amazon has a pretty good checklist for how to do this. You need X.509 certificates so only you and Amazon can access your AMI, which for most layers is a sensible security precaution. For this walkthrough I’ll assume you have the private key downloaded to ~/certs/pk-X509.pem and the the certificate downloaded to ~/certs/cert-X509.pem.
On the server itself:
1
|
|
On your local computer:
1
|
|
This will securely transfer your certs up to the server. Make sure to replace your-user
with whatever user on OpsWorks you have permission to access, and your-servers-public-dns
with the public DNS record for your server.
I’m not totally sure why the AMI tools and API tools are separate packages, but since they are you’ll need to install them individually. For the AMI tools:
1 2 3 4 5 6 |
|
If you got a different version of the AMI tools than 1.5.3, you’ll want to replace the AMI tools directory with the proper version.
A similar process to step 3.
1 2 3 4 5 6 7 8 |
|
Again, if you downloaded a different version of the API tools, you’ll need to change the API tools directory. Also replace YourAccessKey
and YourSecretKey
with your real access and secret keys.
On Ubuntu 12.04, I didn’t have to do anything except this:
1
|
|
But if your image has boot problems GRUB is the most likely culprit. Amazon has a good walkthrough of how to set up legacy GRUB properly – following it should correct any boot issues you experience on your AMI.
Make sure everything and anything is stopped on the target server. A non-exclusive list:
1 2 3 4 5 6 |
|
Running services can destroy the integrity of the image. Make sure everything is stopped before you waste your time!
All of the instance-specific config directories must be destroyed, or OpsWorks will fail to provision the new image.
1 2 3 4 5 6 7 |
|
Finally, after all that setup, you’re ready to actually bundle the volume.
1 2 3 4 5 6 |
|
Note that we provide the certificate locations as part of this command, so if your certs are named differently change that name above. Also you need to provide the account ID number for the -u flag. You can find this on your security credentials IAM page, or if you need more help, check out Amazon’s documentation on finding your account ID number.
This command will probably take a long while to run.
Once the volume is bundled, go to S3 and create a bucket to receive the machine image. Then run this command on your instance:
1 2 3 |
|
Replace bucket_name
and image_name
with the bucket you created in S3 and whatever you’d like to name the image, and the region with whatever region your bucket is located in (and where you want the AMI to be registered). This will also take awhile to run.
Only one step left, and this is an easy one! You can register the AMI with this command:
1
|
|
You should now successfully see your new image in your list of registered AMIs for your region. Change your layer settings to use a custom image and select the AMI as the image for a new instance and try it out. Hopefully you’ll have just cut out a fair amount of time from your instance provisioning process.
]]>Rather accidentally, actually: I fell into it. I’ve been a software developer for about a decade now, since I graduated college, and I’ve always loved it – so much so that I couldn’t put it down when I got home. I was always programming, either on personal projects or side gigs outside my day job. Eventually those gigs got bigger and bigger, and my friends and clients referred me to other people… for awhile, about two years ago, it almost felt like I was working a night job that was almost exactly the same as my day job. Then the startup where I was working full time ran into money problems and had to cut my hours pretty dramatically. I always intended to find another full time position, but I just got so busy with my client work that I didn’t have the time: and that was the start of Symonds & Son.
“No, don’t!” They hated the idea. I’d mentioned it before – I don’t like having a boss, and I’d frequently wondered what it would be like to work for clients, rather than managers. But for my father, consulting has always been very boom-and-bust: there were times he was so busy he was hardly home, and then times he’d be sitting around being unhappy. My parents told me to find a stable, full-time job at a big company like Apple, or even a smaller company where I wouldn’t have to worry so much about the future. But I had an idea to make my own company and I wanted to do it, and when they saw they couldn’t convince me otherwise they gave me advice on how to make it work… and eventually I managed to convince them to join me!
My dad.
I’ve always relied on my dad to help me do contract proof-reading and negotiations. He’s had a lot of experience doing business development and his advice has always been helpful… though, to his annoyance, I haven’t always followed it. Recently my father’s business hasn’t been as busy as he liked, but meanwhile mine has been growing really quickly, and I’ve needed a lot more help than usual on the business end. Bringing him in was a natural fit.
Mom started helping me out during the wedding, so about six months ago – she was our wedding planner. She organized speaking with florists, tasting food, talking with photographers… and quickly I found her help (personally and professionally) indispensable. She’s my personal assistant and helps me keep focused on my business by taking care of invoicing and whatever else the company needs.
So much! It’s really different than I thought it would be: I imagined myself programming day in and day out, but there’s a lot of overhead involved. Running a business with employees and partners is like two jobs. In the first one, I’m programming and making the product my clients want, and in the second, I’m organizing, marketing, selling and negotiating. But I don’t mind all the work… in fact, I love working with my family on a day-to-day basis. Even if Symonds & Son isn’t around in 10 years I’ll have great memories of the amazing things we accomplished together – but I hope we’re around even longer than that.
On average, really good. Since college I’ve gotten along with my parents quite well, and I think since I’ve grown into an adult we’ve gotten along better. Our relationship during high school and even during the beginning of college was a little bit of a mess, but now that we’re all adults we can actually be friends, which I think works out really well.
Yes, a thousand times yes. My father and I spent most of my high school not understanding each other and butting heads. We’d get into arguments constantly, and I think that’s part of the reason I don’t listen to his advice even when I should: I spent most of high school not listening to him and it seemed to work out pretty well. And mom was always pushing me, to get better grades, to do more extracurricular activities, to practice the flute more… I think that if I told myself in high school that I would literally pay her to keep doing that to me as an adult, I would’ve laughed until I died. But I needed help, and I knew they were competent, skilled people who could help me. So, as I said, it just seemed like a natural fit.
That I don’t listen to them! They definitely want me to communicate with them a lot more than I feel comfortable with. They’re always calling and checking up on me and my husband, both personally and in regards to the business, but I have a very full schedule at the best of times: it can be exhausting to satisfy their demands. And from their perspective, I’m not nearly responsive enough. I don’t return calls promptly or deliver them the information they need to do their jobs in a timely manner.
The primary process difference is between me and my father. Dad is very dramatic, as I’ve said before: everything is a crisis, whereas with me, I’m usually calm and collected. So when dad is yelling at me that some work must get done, I quietly tell him that it will, and he goes bananas because he thinks I’m not taking it seriously, and I can’t believe he’s getting so worked up about it. Both of us get our work done in the end, but the meeting of the two processes can be a little difficult, and usually involves a lot of my father hanging up on me and swearing to my mom that he’ll never talk to me again.
Our personalities are very different, though I touched on that in your last question. My mom and I have more similar dispositions, and I’ve seen her handle my dad expertly for years. I know there’s a lot of bluster, but it accomplishes a lot too. He’s passionate and a really skilled negotiator: he’s done some impressive stuff on our conference calls with the same righteous attitude that can get on my nerves. And I know I have that anger in me too – I’ve definitely had moments where I feel like I’ve had to channel my father to get my way. Few people can resist Steve Symonds head on.
I also feel like I’m much more placid at work than I am in my personal life. I try to be clear-headed and visionary in my role as president of the company, but outside the office I cut loose a lot more. I tend to act the same around my parents though, so maybe my “president of the company” personality and my “son” personality are actually pretty close.
I’d say the advice is about half solicited, half unsolicited… And I usually weigh his advice very seriously. That doesn’t mean I take it though! Dad has been very successful in business, but I feel like I’ve been successful too, with many great things still in my future. If I want Symonds & Son to be the best company it can possibly be, I need to hear his advice, compare it to what I think the best course of action is, and then make a decision… and that’s how I’ve used his suggestions in the past.
Of course this drives him bonkers, but I covered our process differences earlier.
Calls me all the time! I have meetings all day long, and when I’m not in meetings I’m programming. I don’t have time to talk on the phone. My employee, when he wants to talk to me, IMs me, which is much more convenient. I need to get my parents to start doing that too. And as I mentioned, the drama aspect.
Nags me, which ironically is what I pay her to do! It can be very difficult for me to realize that she’s just doing her job when she bugs me to do stuff, especially considering how she used to do it all the time for free and drive me up a wall. If I’m already very tense, it can be difficult not to snap at her when she’s reminding me about my obligations.
]]>Of course, I’ve probably obliterated any potential savings by spending so much time experimenting, but now that I’ve done it you don’t have to! Listed here is everything important to me to get my development environment zipping along. Hopefully you’ll be able to find something useful here too that makes your programming experiences a little bit faster.
Font: Source Code Pro (Free) A good, clean, monospaced font is extremely important. You’ll be staring at it in your text editor and on your terminal, and it has to be easy on the eyes while still differentiating similar-looking characters (l as in llama, 1 as in the number, I as in yours truly). I’ve used Inconsolata, Monaco, and Anonymous extensively, but found Source Code Pro to be superior in almost all respects: more readable, more beautiful, less tiring to the eyes. Realistically they’re all pretty close though, so just find something you like and go with it.
Computer: MacBook ($Lots) Ruby is cross-platform: you can develop on a Mac, Linux, or even Windows if you like. I prefer a Mac. They’re extremely reliable, very well-designed, with rock-solid performance characteristics and extremely impressive support guarantees. They really only have one downside, but it’s a doozy: the big price tag. See if you can get your employer to buy one for you. Some people don’t like OSX, but in my mind it unifies a good user experience with the power of Unix under the hood. There’s definitely some stuff about Windows I like more (I have a gaming PC that uses Windows 8), but for development, Macs can’t be beat. They also have great, first-class tool support for anything you’re likely to do in Rails.
Shell: zsh, specifically oh-my-zsh (Free) zsh is what bash wants to be. Command completion, sharing of history between shells, spelling correction, amazing prompts… and oh-my-zsh makes it easy to set up and install. If you need some inspiration for your .zshrc
, check out mine! My favorite feature is sleeping 10 seconds after you do an rm
glob… this has saved me from my own stupidity more than once. (Lots of stuff from my .zshrc
is taken from this StackOverflow question.)
1Password ($50) Insanely expensive compared to most other software on this list, but also pretty non-negotiable. I have over 300 passwords stored in it – including six client logins to AWS, four to RackSpace, and untold multiples for my own servers: how am I supposed to keep all of that straight myself? And generate new ultra-secure passwords whenever I need? Well thankfully I don’t have to. 1Password has bulletproof security and I’m never concerned I’ll lose, forget, or even care about any individual password ever again.
Alfred (Free to try, £17 to buy) I use Alfred to open nearly everything on my computer. Not just apps! Alfred can open web pages, login automatically with 1Password integration, perform calculations, run one-off terminal commands, find and open files… Alfred has enormously simplified everything I do with my Mac. Definitely a huge time-saver.
Asepsis (Free) So many important files on your comptuer are dotfiles, which are hidden by default on Macs. So you turn on viewing hidden files… and suddenly, everywhere you go, you see the dreaded .DS_Store
. I hate seeing .DS_Store
files. They appear in every directory, they appear on your desktop, they appear in your nightmares… well Asepsis gets rid of all of them. And it’s free.
Bartender (Free to try, $15 to buy) Look at the upper-right of your Mac. Do you see the volume icon there? The AirPlay icon? Even worse, your own name? (What, are you worried you’ll forget it?) Banish the clutter into Bartender and clean up your menu bar: you can hide all the icons you never (or rarely) use, and even configure them to display only when their app does something important.
Caffeine (Free) Prevent your Mac from ever falling asleep. Helpful if you’ve ever started a long-running task, left your computer for awhile, and returned to find it had stopped to take a nap.
CloudApp (Free, or $4.99 a month) CloudApp gives you drag-and-drop sharing of any file you’d desire. I love having my drops available at a vanity URL (http://veratic.us) – you’d be surprised how many developers say how cool that is, and ask how they can get one of their own. If you don’t want the URL then the free version should suit you just fine.
Flint (Free, then $14.99) I love Campfire and BaseCamp. Flint makes Campfire easy-to-use. Even better, it gives me a badged icon in my dock when people speak, and a growl notification when they mention my name.
Gistify ($1.99) Shameless self-promotion, but I honestly love the app I made for myself! I make gists all the time; gistify gives you drag-and-drop gisting of snippets or files, and allows you to create gists from your clipboard contents. Super convenient and super lightweight.
Growl (Free, or $3.99) The Apple Notification Center is pretty good… but Growl still seems to be the gold standard for notifications. One of the first pieces of software I install on a new computer. (That said, I have no idea what the $3.99 version does. Growl is one of the few pieces of software on this list I’ve never bought.)
HyperDock (Free, or $9.95) Allows you to hover over your dock items and see a preview pane summarizing all of that app’s open windows, a la Windows’ default behavior when you hover over an app in the taskbar. Extremely helpful if you have a lot of files open simultaneously – for example, if you have lots of projects open at once in your text editor.
iTerm 2 (Free) The best terminal replacement ever – I don’t know how I ever lived without, and once you start using it, you’ll never be able to go back to the default Mac terminal. It has so many nice improvements I can’t even list them all, and for the absurd price of “free” there’s absolutely no reason not to use it.
Little Snitch (Free to try, $34.95 to buy) Little Snitch is a really excellent software firewall. I’m incredibly paranoid about network security, especially on my development machine. Little Snitch allows me to opt-out of connections I don’t trust and really lock down my computer – not only in general, but on a network-by-network basis. As a bonus, it lets me see my network traffic so I get notified if DropBox suddenly decides to resync everything.
Monosnap (Free, $4.99 to enable sharing) The best skitch replacement (since Evernote ruined it). Fast screen or window captures with built-in DropBox, CloudApp, and Evernote uploading for only $4.99. It’ll record videos of your screen and even take selfies on your Mac’s camera, if you’re too lazy to grab your phone.
Sublime Text 3 (Free, then $70(!)) My favorite text editor (and I’ve tried most of them). Expensive? Definitely. But you get what you pay for. ST3’s best feature is its incredibly versatile package system – there’s a package for pretty much everything, and as a bonus it supports most of TextMate’s plugins as well. Despite its depth and power, it’s still light-weight enough to start instantly and doesn’t spend any time doing reindexing or otherwise chugging. ST3 is also cross-platform if you end up doing a bit of light programming on something other than your primarily development machine, which is useful for standardizing your editor everywhere.
TotalSpaces (Free to try, $18 to buy) I don’t like Mac’s default Spaces implementation – four desktops right next to each other means the furthest space could be two screens away, and the widgets view is totally worthless. TotalSpaces lets you stack desktops in a grid and move left, right, up, and down. Now every space is only one swipe away from any other space. As a bonus you can get rid of the widgets as well.
Tweetbot ($19.99) Twitter is actually a valuable professional tool for me. It lets me keep on top of interesting news and developments in the tech world, contact authors of the software I use directly, and find lots of pictures of kittens. But leaving the #kittens aside, Tweetbot is the best app I’ve used for twittering. It’s fast, effective, and stays out of my way when I don’t want it.
Unibox ($9.99) I like Unibox compared to Mail.app and Mail Pilot: it’s speedy, it organizes mail effectively, and I feel like I have a bit more control of my inbox. That said I haven’t yet run into the program that completely replaces Gmail in a browser for me… Unibox is as close as I’ve come and I use it on a daily basis, but if I’m searching for a specific piece of mail, I turn to gmail.
If you love Sublime Text 3, you’ll want the best packages for it too. Here’s what I recommend:
Package Control allows you to find, install, and update Sublime Text 3 packages. A prerequisite for installing anything else without going insane.
Soda is a beautiful, functional theme that I really enjoy. I’ve tried a few others and didn’t find them nearly as compelling (SpaceGray is also quite nice, though the syntax highlighting isn’t to my taste). Because I find it easier to read light text on a dark background, I use Soda Dark specifically.
Alignment makes it easy to align everything properly and make your code nice and readable.
BracketHighlighter shows you if you’re inside brackets (or a block, or HTML element) and where the start and end of said brackets are. Invaluable.
git gives you git integration right in Sublime Text 3, and works great with…
GitGutter, which shows in your gutter changes that are waiting to be committed.
SidebarEnhancements gives you tons more options when you right-click anything in your sidebar, including clipboard, move to trash, reveal in finder, open in another program, and everything else you’d expect a sidebar to reasonably contain.
SublimeLinter (and linters for your languages of choice) provides cheap and easy sanity-checking of your code. Now you’ll know as soon as you save (or as soon as you finish typing, if you like) if you fat-fingered a variable name.
homebrew is the best package manager I’ve come across. It’ll install everything from libxml2 to postgresql from source automatically, helps you manage upgrades from one version to the next, and is fast and performant.
rbenv allows you to have multiple versions of Ruby on your system simultaneously, each without stepping on the others’ toes. If you regularly develop in runtimes other than MRI (I have to do JRuby stuff somewhat frequently, for example) this allows you switch back and forth without going insane.
zeus preloads your Rails apps so that tests, the console, and the server all load lightning-fast. The milliseconds you wait for Rails to start now become microseconds, saving you entire minutes per year… And isn’t that really the point of optimization?
I’m sure there’s a lot that I missed and some things I don’t even know about. If I didn’t include your favorite app, Sublime Text plugin, or Rails development tool let me know @Veraticus – there’s nothing I love more than taking a new app for a spin and even adopting it if it turns out to be amazing!
]]>Symonds & Son isn’t named to sound cute and hipstery. I’m the son; I’m in business with my father, and I recently hired my mother to be my assistant and bookkeeper. The company’s name is intended to evoke images of traditional, reliable family businesses, where trust, excellence, and a reputation for quality products are their cornerstones. These companies, traditionally centered around carpentry and other hand-made professions, are my models for how to run Symonds & Son. We are, if you’ll forgive me using the phrase, a bespoke development shop, where our great output and superb relationship with our clients are our highest priorities.
But building my profession into a company – and a company with my parents, no less – hasn’t been an easy road. There have been a lot of compromises we’ve had to make to get this ship sailing straight. Here, I’ll speak to how I made this new parent/child relationship work… and how I hope to continue making it work for years to come.
Let me be perfectly honest here: I was not a model child. Quite the opposite, in fact – I was the picture of a rebellious teenager. I was surly, I hated authority, I refused to leave the basement and I would play computer games constantly in lieu of doing homework. My parents and I would have huge fights that would last for hours, typically centered around my grades, and I would go screeching off into the night (in their car, of course) to hang out with my friends at Dunkin Donuts and whine about the many injustices of life, parents, and school.
(I was not a great student.)
My relationship with my parents really did not improve until after college. Though I never liked school, I found my niche in the workplace: arbitrary grades meant little to me, but performance tied to money? Now that was a formula I could understand! As my career flourished my relationship with my parents got better. I like to think that they gradually grew to respect my skills and ambition, but more realistically I think they were just happy that I had launched successfully when so many of my peers had been burned and left un- or under-employed by the poor job market.
My father, a successful consultant in his own right, always advised me never to consult. Every time I eyed doing hourly work, he told me to firmly refocus my sights on a full-time job. He’d speak to the benefits, the retirement account, the possibilities for advancement big companies could offer – and, of course, I completely disregarded his advice. I found employment first with startups (not exactly bastions of job security or big 401ks) and then freelancing.
So if you want to take one piece of advice from my post, it’s this: if you want to go into business with your parents, ruthlessly ignore their recommendations at every turn!
My father’s niche had always been contracts, contacts, and business development, areas in which I am admittedly weak. He’s helped multi-million dollar companies divide and conquer their industries, and find partners to allow them to do so. When I told him I was dead set on making this consulting shindig work, he responded that he’d be available if I needed his help. Thus I found myself relying on him more and more… first to read contracts, then to perform light business development, and finally to properly structure my own growing business and best serve my clientele. In the end I suggested we just incorporate together. He agreed, and thus Symonds & Son was born.
Neither my father nor I are accustomed to knuckling under in arguments, and so our new business relationship has hardly been problem-free. We have some very different perspectives on the proper ways to interact with clients, communicate on a daily basis, and turn single contracts into recurring revenue. For example, I hate answering the phone during the day, as I’m usually busy coding – but he can’t stand getting directed to voicemail when he knows that I’m around and just not picking up.
Our solution for that particular problem is better communication and faster turnarounds. He knows he can get ahold of me if there’s an emergency, and otherwise that I’ll return his call as soon as possible… and in return, he texts me a lot more, which allows me to respond when I’m available and in my own time.
But definitely the most important lesson I’ve learned is that my father is indeed an expert in what he does, and generally knows what he’s doing – and that’s the same lesson I try to impose on him when he starts stepping on my toes too much. It can be difficult to give him the reins during delicate business operations, but I actually find it helpful as well. Like most entrepreneurial people, I love being in control and feel most comfortable when I’m managing as many of the aspects of any situation as I can. But other people are great at what they do too, and they can only prove it when given the chance to shine.
Or, more pithily, if I’m to succeed personally, I have step back and let the people I trust succeed. Only if I allow it to happen can it happen.
As my business expanded, I found I had less and less time for simple business tasks – invoicing, talking to my accountant, setting up and managing a corporate bank account… My to-do list was growing longer and longer with absolutely no hope of me ever breaking free to accomplish any of it. After all, if I stopped coding then I stopped getting paid, and consulting is not exactly a business that lets one rest on their laurels.
I needed an assistant.
My mother had handled all our scheduling, accounts, and priorities growing up: she was a successful businesswoman in her own right as well, though she left the corporate world and went into non-profit when my sister was born. That changed in the beginning of 2013 when my parents moved. Mom suddenly found herself with less on her plate than when she lived in Connecticut, and it was difficult for her to translate her great non-profit experience in the suburbs of Connecticut to the non-profits in the heart of Chicago.
Growing up, I hated being nagged. But I realized what I needed was an incredibly prioritized, driven person to help me with the day-to-day operations of my business… and my mother had time to spare and thirty years of experience managing me and my time. As much as I hated to admit it, I needed her to really make Symonds & Son a success.
And so far it’s been working out really well – almost scarily well. My mother is an efficient woman, and when I give her a list she turns it around in a timeframe I find just astounding. For my part, I’ve learned to respect her gentle nagging: not only does she have my best interests at heart, but the best interests of our company. She’s working on behalf of the company to make it more efficient. If I stood in the way of that, I would be a bad employer, and while I might not have been the best son in the entire world, I at least want to be a pretty good boss.
I never turned to my parents with charity. They are successful, independent people in their own right, and I am lucky they’ve decided to invest their time and energy in my consultancy. But there are lots of people out there who would want the opportunity to draw a paycheck from Symonds & Son, some potentially even more qualified than mom and dad. So why did I decide to go into business with them?
Partly it’s where they were in their lives: they had the time and energy to dedicate to a new venture. But more than that, it’s that I already know, respect, and trust them. I want the values of my company to reflect the values of my family: a firm commitment to client work can only really be achieved if I’m not constantly watching my back for a dagger, or if I’m free from worries about the honesty and priorities of my employees.
I could have gone out, interviewed candidates, selected potentials, hired a few, and eventually learned to trust and respect them. But I had excellent, well-qualified, trustworthy and experienced candidates I already had a good relationship with. To answer my earlier rhetorical question with another, why should I go searching for what I already had?
Of course, I would be disingenuous if I said I was never concerned. What if I had to have some sort of Serious Conversation™ with mom and dad about their performance, or salaries, or any of the other number of difficult conversations any company might have to have with its employees? Or, to get to the heart of the matter: could I really fire my own parents if I had to?
I like to think it’ll never come to that. Symonds & Son is a family business, and we’ll succeed (or fail) as a unit. If the worst comes and I’m forced to close up shop, then it means unemployment for all of us – at least temporarily. It won’t be the end of any of our careers. I’ll land somewhere else doing Rails, most likely, and my father will pick up other clients, and my mom will find a non-profit. It’ll be another chapter in our lives closed, but hopefully an overall positive one.
Certainly working together as a family has been a challenging experience, but it’s brought me closer to my parents in a way I wouldn’t have imagined even a few years ago. I’ve learned to respect them for their talents and abilities, rather than as caregivers with the patience to raise me and my sister. I’m not saying this kind of business is for everyone – maybe it works only for us, and even then, who knows if it’ll work forever? But in trying to model Symonds & Son after traditional, family-run businesses, I’ve found capable, competent employees right under my nose: and my clients seem to enjoy working with us, so we must be doing something right.
And I would be remiss if I didn’t include at least a little advertising here! If you’re interested in seeing Symonds & Son work for you, we make amazing software products and scalable Rails solutions. Get in touch and see what a family business can do for you. You might just be surprised at how efficient and effective the new family business can be.
]]>At the start of 2013, I was a salaried employee working at a startup. The rate was fine, but I spent a lot of time working – long hours in the evening to make aggressive sprints, and many meetings during the day to discuss development priorities and investor relations. I wasn’t happy, but it was a job, and I was satisfied with it.
A few months into my tenure, I was offered a salary adjustment to help make my company’s bottom-line more attractive. Instead of taking it, I proposed an alternative arrangement: I would become a consultant at a rate very similar to my old hourly and I’d work part-time. After some negotiation, a deal was struck, and I was officially a free agent.
But I wasn’t happy – indeed, quite the opposite. I spent weeks beforehand freaking out. I had the extra hours available to make up my cut income, but I’d need clients to actually pay me for that time. Otherwise I’d need to find another salaried position, and fast, since I didn’t have much in the way of savings. Disaster was looming, and I spent sleepless nights trying to figure out how I’d find a client, how I’d convince them to actually pay me, and, if that failed, how I’d explain a gap on my resume to future potential employers.
Yet almost immediately upon taking the plunge, an old client wanted me to do new work for them. They recommended me to another company, who told one of their clients about me, and very shortly after I became part-time, I was full-time working for my clients. It took me almost all of 2013 to understand what had happened.
In case you didn’t know, Rails developers – specifically, good Rails developers with experience in modern tools, an interest in improving themselves, and an aggressive talent for development – are extremely hard to find. In fact, finding developers who are capable of programming in their chosen language at all is a challenge.
But why accept a generalized statement when I can give you a specific example? I’m presently working with a client to vet Rails engineers. The rate they’re willing to pay is really quite good, yet the candidates their recruiter finds are just terrible. This might be a topic for a separate post, but of the dozen people I’ve interviewed:
Yet the recruiter says these people are snatched up all the time, at a rate of roughly $100 an hour. I honestly have absolutely no idea how this happens. I’m not exaggerating even a little bit when I say I believe that none of these people can code at all – they have no GitHub profiles, no code samples, absolutely nothing to their names.
(Incidentally, can you do better than these idiot candidates? Let me know, I have a pile of money with your name on it.)
I’ve come to believe that these people are why I’ve succeeded. If you bring dedication, honesty, and actual, real skill to your clients, they will recognize your contributions, keep coming back to you, and tell all their friends about you. And they’re willing to pay your hourly rate for long nights, excessive meetings, and even just listening to their plans and helping them improve their processes.
But if that’s so, then why do most engineers, even the good ones, stay put at their full-time, salaried positions?
I think a lot of it has to do with the startup culture in San Francisco presently. Weirdly, I think it has the effect of keeping engineer salaries artificially lowered.
Our industry is dominated by talk of the tech bubble: all the press is about acquisitions, huge seed funding rounds, and successful entrepreneurs’ new projects. But the amount of money in Silicon Valley is really quite limited. There are enormous industries out there that need skilled programmers but lack the sex appeal of a startup or coverage in TechCrunch – yet they have applications in Rails and backend infrastructure needs too.
And they also have way, way more money. Most of my clients are not extremely large businesses in their fields, but the amount of capital they have dwarfs that of even established startups. And they actually have business models that have worked for them for many years, so I worry less about revenue stream issues (or, heaven forbid, them folding overnight).
Even better, to these companies, you aren’t just an engineer with a set salary: you’re solving a business problem with software. Your value to them is measured in the millions of dollars you saved their company, not the amount they’re expecting to pay to a Rails engineer. And by charging on the former, not the latter, you can turn a very tidy profit indeed.
Of course, the process of getting my business up and running hasn’t been all sunshine and roses. I’ve made some mistakes and wished I’d handled a few things differently.
I nickled-and-dimed a client on change requests, alienating that client and making myself appear less professional. Said client did not have a whole lot of money, and while the initial contract amount was commensurately very low, she really didn’t appreciate me charging additional for some very minor changes. I should have just sucked it up and done the work, leaving both of us with warm fuzzes in the end, even if I took a slight loss on the contract. Most of my clients hear about me from other satisfied clients, and I would have been better served by her loving me than making a little more money.
For pricing my services, I need to start high and work my way down. I generally start client conversations on my hourly rate at what I would consider a reasonable ultimate number, and then allow myself to be driven down from there – generally because the client wants a long-term contract and expects to save on my hourly based on the length of the engagement.
More projects, less hourly. When starting as a consultant, I was really selling only my hours. Now Symonds & Son is a business in its own right, and I’ve hired designers and developers to help with my workload. Working with other talented individuals makes much more sense on a project basis, where I can package their (and my) hours together.
“Shut up already Josh, I think this is a great idea and want to become a consultant too! What’s next?”
Make sure you have a good track record and established public credentials. Verify your friends (and your ex-employers) will vouch for the quality of your code and the quality of you as an individual. Go to meetups, write blog posts, have open-sourced code on GitHub – the more stuff you have on record, the better. Your clients will want to know everything about you they can before they even meet you.
Have a backup plan in case everything goes wrong. Mine was “find another full-time Rails job,” something that I’ve never traditionally had a problem finding. At least think about a safety net so that taking the plunge is less scary.
And consider if you really want to be a mercenary. Many talented coders I know work for peanuts, but they do so for non-profits, amazing startups, and benefit corporations. They don’t care that they’re not taking home tons of money; they are making a difference in the world, which matters more to them than any paycheck ever could.
But if the idea of consulting appeals to you, then I encourage you to take the plunge. As engineers, our services are as in-demand as ever; if you are a competent engineer, you can turn your skills from a salary into a solution, and companies pay much more handsomely for the latter. And if you try to fly and fail, you have a good chance of landing at another job anyway – so really, the risk is pretty minimal. And if you take off and soar away, let me know: I always need exciting new companies to work on big contracts with!
]]>So I decided to make it.
cryptolalia is a Ruby gem that allows the creation of arbitrarily-complicated ciphertexts from plaintexts. At release, it has the following ciphers:
You’ve probably seen most of these ciphers before. But they become way more difficult to decode if you choose weird alphabets, Victorian-era source texts, and long-dead numeral systems. Combine these ciphers with excellent placement and you too can take the cryptography world by storm!
Taken from the cryptolalia README:
The above image contains a ciphertext! It was inserted there using cryptolalia in the following manner:
The plaintext (“secrets are fun”) was transformed with a Pollux Morse code cipher:
1 2 3 4 5 6 |
|
The result of the Pollux cipher is fed into a Beale homophonic substitution cipher with the Declaration of Independence as a source text:
1 2 3 4 |
|
The result of the homophonic substitution cipher is further moved into a steganographic PNG cipher to encode it into the least-significant bits of an image:
1 2 3 4 5 6 |
|
Don’t believe me? You can decode it yourself, also using cryptolalia:
Download the file above (rainbow.png) locally and decipher it with the steganographic PNG decipherer:
1 2 3 4 |
|
Insert the numbers of the Beale homophonic substitution cipher back in:
1 2 3 4 |
|
And finally, plug it right back into the Pollux cipher:
1 2 3 4 5 6 |
|
This can sometimes be a good thing. It’s always touching to receive a heartfelt letter of thanks and admiration from a customer. But much more commonly, you’ll hear complaints, demands, and anger.
Under a deluge like that, it’s easy to become disillusioned and bitter. I see it all the time: programmers who hide in their shells at the slightest provocation. They make excuses, shift blame, and refuse to own their actions. I think this attitude is ultimately cowardly – if you’re willing to take credit for the good things you’ve done, you have to “take credit” for the bad stuff too. And cowardice of this sort contributes to making bad code, since your goal is no longer to to solve your organization’s problems, but to shirk responsibility.
That’s what I call coding defensively – an entirely separate subject from defensive programming. Coding defensively mires you in your codebase. It’s hard to go forward because you don’t want more responsibility, and you can’t go back because it would be admitting error. And it’s a self-reinforcing cycle: once you start becoming defensive, it’s difficult to stop.
I was a defensive coder too. I struggled to own the results of my actions: I would refuse to discuss solutions in a sensible, level-headed way. It took me a long time to change my approach, and as part of doing so I wrote a list of maxims summarizing how I wanted to think and act. When I feel myself slipping I go over them to refresh my resolve. I’m publishing them here in the hope that they’ll help someone else as much as they’ve helped me.
First, you have to realize that people – even you – make mistakes. It’s not productive to pretend that it didn’t happen or that someone else is at fault: if you know in your heart that the responsibility lies with you, then don’t try to shift blame. It was your mistake, and part of being a human is making mistakes, so own that error with all your heart!
I found that achnowledging my fault and being accepting of it removed a lot of the impetus to be defensive. I no longer felt as ashamed of what I’d done or that I had to spend time and energy making excuses. The “excuse” is that mistakes happen: what matters now isn’t that the mistake occurred, but how you’ll go about addressing it.
Please don’t get me wrong – I’m not saying you should be making mistakes left and right flippantly. Avoid them if you can, through proactive testing, code reviews, and sensible QA. But they’ll still happen, even if you’re very good. Since you can’t change whether or not you’ll make mistakes, you may as well change your attitude to them.
In the face of anger or blame, don’t turtle up! Instead, kill them with kindness.
As I said way back at the beginning of this post, what we do naturally inspires passion in those around us. Frequently when people run into an engineering problem, this passion will turn to anger. By the time the problem has percolated to us the anger has found its target: the programmer responsible for creating the code in the first place.
Friendliness reminds the people you’re interacting with that you’re just as dedicated to a solution as they are. Resist when they try to brand you as an antagonist. Their reaction is natural because they believe you made the problem: but in fact, you’re partners together in trying to solve it, and kindness reminds them of this fact.
You’ll be surprised at the turnarounds you see in people’s attitudes when you’re unrelentingly friendly to them in return. And once the anger has dissipated, you can work towards a resolution.
Or, if you’re a Glengarry Glen Ross fan, “always be solving.”
Putting your existing solutions on a pedestal isn’t productive. There’s no inherent value to what you have other than that it works – and sometimes, not even that! Hand-in-hand with this, concentrating only on the potential downsides of new technology blinds you to potentially tremendous upsides.
Dismissing solutions out of hand – particularly if you don’t understand them or it seems like they’d be difficult to implement – is an easy, natural reaction. But what your stakeholders are searching for might exist outside your comfort zone. It might involve using a piece of software with which you have little experience, or, in the worst case, might be implemented in a programming language you don’t even know.
If it’s still more or less perfect despite those flaws, then you have to go for it. Your responsibility is to do what’s right, not what’s easy.
And guess what? Afterwards, you’ll probably feel way more comfortable implementing that solution again in the future, and professional growth is always pretty awesome… and leads to personal growth as well.
Even in the face of overwhelming negativity or horrible problems, you have to remind yourself that your overall effect is a positive one. As humans, we spend an inordinate amount of time concentrating on negative experiences: in the past, perhaps, an embarrassing moment in high school that everyone but you has forgotten, and in your day-to-day job, the errors and problems you feel largely responsible for.
It can be helpful to remind yourself of the good things you’ve done too, and that, in the balance, the good outweighs the bad. Doing so gives you the courage to hold your head high, and gives you the self-confidence to tackle your problems head on, instead of hiding from them.
And that willingness to embrace reality is what prevents defensive coding. In the end, your willingness to act – and the attitude you have towards action – can mean more to people than what you do. By trying to be a proactive, positive force of good, you’re telling the people who care about your code that you care about it as well. More than anything else, that will garner their respect and admiration and lead them to believe you’re coding for them, not coding defensively.
]]>Based on my experiences, being a great programmer has nothing to do with age, education, or how much money you make. It’s all in how you act and, more deeply, how you think. I’ve noticed a consistent set of habits in the programmers I admire. More than knowledge of their chosen language, deep understanding of data structures and algorithms, or even more than years of on-the-job experience – the way they communicate, the way they conduct themselves, and the way they approach programming speak volumes as to their amazing level of skill.
Certainly there’s more to being a good programmer than anyone can enumerate, and I wouldn’t judge any programmer solely based on the presence (or lack) of these practices. But I do know wisdom when I see it, and when I see a programmer expressing these traits, I think, “This person really knows what they’re doing.”
Or “measure twice, cut once.” Or “Google it.”
Whatever you call it, most programming problems you’re likely to encounter have been solved in some form already – Ecclesiastes had it right in observing there was nothing new under the sun. In the form of libraries on GitHub, blog posts on the Internet, or just getting in touch with experienced individuals, good programmers know to do the research before they try to solve a problem.
I’ve seen even great programmers rush to solutions: but the worst programmers I’ve worked with never consult others, and as a result do huge amounts of duplicate work or tackle issues in precisely the wrong way. And unfortunately, they end up paying for their mistakes later.
This includes desymbolicating stack traces. Yes, it’s ugly and unfortunate – but if you’re not willing to do it how are you going to know what’s wrong? The most effective programmers I know aren’t afraid to dig deep into problems. The least effective see there was an error but don’t even bother reading the error message. (This sounds ridiculous but you’d be surprised how frequently I run into it.)
To take this a step further, great programmers see a problem and hunger to solve it. Reading error messages are for them just the beginning: they’re anxious to dive deep into the issue that it represents and figure out what’s going wrong. They’re not interested in passing responsibility, they’re interested in finding a solution. The buck really does stop with them.
Documentation, tests, people: these things lie. Not intentionally, but if you want to know exactly how something works, then you must view the source yourself.
Don’t be afraid if it’s not a language you’re super familiar with – if you’re primarily a Ruby programmer and you suspect there’s an error in one of Ruby’s C libraries, go crack it open and have a look. Yes, you might not get it. But who knows? You just might, and you stand a way better chance than if you didn’t try at all!
If you’re in a closed-source environment then this becomes a lot harder, unfortunately, but the principle still applies. Poor programmers have little interest in viewing source, and as a result problems flummox them much longer than those who are willing to do some research.
Good programmers are driven to act. There’s a compulsion in them that can be difficult to control: they’re implementing a solution as soon as they identify a problem or see a new feature request, sometimes too early and sometimes too aggressively. But their instinctive response to a problem is to tackle it head-on.
This can be troublesome at times – but their enthusiasm is a crucial part of being good at what they do. While some might stonewall or wait for the problem to go away on its own, good programmers start solving.
To put it more simply (and, perhaps, too obviously): if you see someone excitedly grabbing problems and acting on them, chances are you have a good programmer on your hands.
This is mostly a bad programmer trait: they tend to bounce from artificial crisis to artificial crisis, never really understanding one before they move to the next. They’ll assign blame to the wrong parts of their program and then spend hours debugging something that works perfectly well. They let their emotions get the better of them, trusting gut feelings rather than careful, explicit analysis.
If you hurry through a problem – or even view each problem as a world-shattering catastrophe – you’re more likely to make errors or not address the underlying issues. Great programmers take the time to understand what’s going wrong, even when there really is a catastrophe: but more than that, they treat regular problems as issues to be solved. Thus they solve more of them more accurately, and do so without raising the tension level of your organization.
Ultimately, programming is a form of communication. Being able to succinctly and eloquently express your thoughts is as crucial in writing code as it is in writing prose – and I’ve always found that those who can write a pithy email, elegant status report, or even just an effective memo will also be better programmers than those who cannot.
This applies to code in addition to English. It’s certainly possible to make a one-liner filled with parenthesis, brackets, and single-letter functions: but if no one understands it, what’s the point? Good programmers take the time to communicate their ideas properly in whatever medium they’re using.
I think this might be the most important aspect of a good programmer (and perhaps the part of this list most generally applicable to fields other than computer science).
If you truly care about what you’re doing – if it’s not just a job, but a hobby, an interest, even a fascination – then you have a major advantage over anyone else in your field. Great programmers are always programming. Their peers will clock out at eight hours a day: not out their job, but of their field in general. They don’t have programming side projects or much interest in contributing back to the community. They’re not constantly trying neat new methods and crazy programming languages just to see how they work.
When I see a programmer who’s doing their own hobby projects on the weekends, who’s involved in the creation of the tools that they use every day, who’s fascinated by the new and the interesting: that’s when I really know I’m looking at an amazing person. Ultimately great programmers view their profession not just as a means to make money, but as a means to make a difference. I think that’s what makes the best programmers who they are. Coding, to them, is a way of shaping the world. And that’s what I truly respect and admire.
]]>Change the NotifierWorker
to look like this:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 |
|
Of course, the big change here is require apn_connection
and the extraction of all the logic that had formerly established our connection with Apple. Now we do that in a new class, sensibly called APNConnection
:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 |
|
The main difference here is that the write
method will raise an error if the connection has become closed – this happens most frequently when you write a bad device token into the stream, which causes the APN service to disconnect you. Frustratingly the closure is detected on the request following the bad request, meaning that a perfectly good request encounters an error for no particularly good reason. The retry code here will attempt to reopen the connection to Apple five times and resend the message, until eventually it gives up.
Using this method I have a robust, failure-resistant push notification service in production that I (and my customers) are very pleased with indeed.
]]>Sprinkle (and rubber) are very different from most other server provisioning software I’ve used – on the one hand, by leveraging Capistrano for server communication (or SSH or Vlad if you prefer), it remains extremely light and focused on just provisioning. But on the other, it inherits most of Capistrano’s downsides too: primary among them is that it’s easy to repeat yourself if you’re not careful. So I wanted to post a quick tip for other people using sprinkle on how to DRY it up just a little bit.
Let’s say you have a helper method you want to include in all policies, packages, and verifiers. Make a module to contain it, something like this:
1 2 3 4 5 6 7 8 9 10 11 12 |
|
The problem is that to use this helper method in a lot of different places requires a little bit of work. You can’t just do something like this:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 |
|
You need to include the Helpers
module in each class: policy, packages, and verifiers. That’s easy enough to do. After you define your helper, do something like this:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 |
|
Now your helpers will be available everywhere you expect, allowing you to use them anywhere but still define them in one place.
I’ll post some more neat sprinkle tidbits in the future, but this by itself allowed me to significantly dry up my code and enjoy my sprinkle experience quite a lot more.
]]>