Demystifying the Rails Asset Pipeline

One of the prominent features of Rails 3.1 was the asset pipeline. The asset pipeline provides some great functionality to rails applications and engines, including dynamic compilation of high-level assets, such as coffeescript and sass, concatenation and compression of asset files in production, and more. To be honest, the asset pipeline was just simply magic to me for quite some time. Recently, I've been slowly peeling back the layers and glancing under the hood, and in this post, I would like to share some of my discoveries.

Put simply, the asset pipeline does a few things:

• Compiles assets such as Coffeescript and SASS down to Javascript and CSS
• Concatenates all Javascript and CSS assets in order to serve less files
• Compresses (minifies) files to be served
• Generates a digest to be added to the file name, allowing the file to be cached indefinitely (if the file contents change, a new digest, and thus a new filename, is generated)

The asset pipeline has several configuration options, and the default configuration in development mode is very different than the default configuration in production mode. To make the functionality of the asset pipeline clear, let's start with what it does by default in development mode.

Development Mode:

The first piece of the puzzle to be understood is the concept of manifest files. A manifest file is simply a Javascript or CSS file which contains some special directives. These directives specify which other Javascript or CSS files should be included when this file is served up. When a rails app is first generated, some default manifest files are generated in app/assets/application.js and app/assets/stylesheets/application.css. For simplicity, let's just talk about the Javascript manifest (everything discussed will apply similarly to the CSS files as well). At the bottom of the file, there are three lines:

//= require jquery
//= require jquery_ujs
//= require_tree .

This means that when we compile this file, we must include the files jquery.js, jquery_ujs.js, and every javascript file in the current directory tree. The files specified here are found by searching the asset path, which by default includes app/assets/javascripts, vendor/assets/javascripts, and lib/assets/javascripts. (Similar paths exist for stylesheets). Now, if you run the rails app and visit the path /assets/application.js in the browser, you will see that the jquery files have been concatenated together, along with the manifest file itself, and any other Javascript files in the directory. This is the basic operation of the asset pipeline.

Another very important part of the asset pipeline is the generation of HTML tags which reference the assets. For example, we can use javascript_include_tag within a view or layout to include a Javascript file in the view. You'll notice that by default, if a view calls javascript_include_tag "application", the generated code looks something like this:

<script src="/assets/jquery.js?body=1" type="text/javascript"></script>
<script src="/assets/jquery_ujs.js?body=1" type="text/javascript"></script>
<script src="/assets/articles.js?body=1" type="text/javascript"></script>
<script src="/assets/application.js?body=1" type="text/javascript"></script>

Two things stand out here. First, there are four script tags where we may expect only one. Second, the URLs have a parameter "body=1". This is because the asset pipeline in development runs by default in debug mode. This simply means that instead of concatenating all the files referenced by the manifest file, all the files will by included separately as above. The "body=1" parameter simply tells the pipeline not to do any concatenation when including a given file, but simply to serve the body of the given file. To see this in action, visit the paths /assets/application.js and /assets/application.js?body=1 and compare the two outcomes.

Furthermore, in development mode, the asset pipeline does not compress the files which are concatenated, or generate digests when it generates the paths.

All of the functionality described above is configurable. To configure the asset pipeline for development mode, open config/environments/development.rb and set the following configuration variables:

• config.assets.enabled - enable the asset pipeline. Defaults to true.
• config.assets.paths - the paths which are searched to find assets referenced by the manifest file. You may add directory paths to this array in order to include them in the search.
• config.assets.debug - turns on debug mode. If set to true, the javascript_include_tag (and stylesheet_link_tag for stylesheets) will include each file in the manifest separately and append "?body=1". Otherwise, they will be concatenated.
• config.assets.digest - tells javascript_include_tag (and stylesheet_link_tag) to generate digests and append them to the filenames.
• config.assets.compress - tells the asset pipeline to compress (minify) all assets which are served.

I would encourage starting up a rails app and changing these configuration parameters to see what they do. It will help a lot in understanding how the pipeline works. Remember to restart the server after changing the configuration.

Production Mode

So that's enough about development mode, how about production mode? Well, as you may guess, config.assets.debug is set to false, config.assets.digest is set to true, and config.assets.compress is set to true by default. However, there is one more trick which is used in production: precompiling!

In production, by default, assets must be precompiled in order to be served. This means that before the app is deployed to production, the rake task assets:precompile must be executed in order to run all assets through the pipeline, generate static Javascript and CSS files to be served, and save them in the public/assets directory. Then, when the app is run, the pipeline is disabled, and when a request comes in to a /assets URL, the precompiled static file is served by the web server. This means that all compilation, concatenation, compression, and digest generation is done by a rake task beforehand, rather than on the fly. This makes perfect sense in production since the assets shouldn't be changing often, and serving static files is a lot faster than compiling the assets for every request.

With this in mind, there are two more configuration parameters to consider:

• config.assets.compile - whether or not the asset pipeline should compile assets on the fly. By default, true in development mode and false in production mode.
• config.assets.precompile - an array of file paths to be precompiled. For example, application.js and application.css are included by default, but if you wish to precompile some other files in production mode (for example, if they need to be referenced separately by an HTML page in the app), then this file's path may be added to the array.

Engines:

One more thing. Rails engines.

Rails engines are wonderful things. Mountable engines essentially give you the ability to have a rails app within another rails app. The inner rails app may also have its own set of assets to add to the pipeline. The good news is, the paths for the engine assets are added to the asset search path, and the engine assets are namespaced. So, if you have an engine called my_engine, it may have a file app/assets/my_engine/application.js. This file may be included into the main app's manifest file by adding the following line to the application's manifest:

//= require my_engine/application.js

In this way, engines no longer need to supply generators to copy their assets into the public folder of the rails app. Instead, the asset pipeline takes care of it for you.


I've learned a lot about the Rails Asset Pipeline in the last little while. I hope you've found this explanation helpful. For much more information and configuration parameters, check out http://guides.rubyonrails.org/asset_pipeline.html

Any comments about the Rails Asset Pipeline? Is it still all black magic?

Tools I can't live without

Hey guys, welcome back to the blog. I hope you're enjoying the content so far, and there's plenty more to come, so stay tuned.

Today I want to give a quick rundown of the tools that I use on a daily basis as I'm developing software. I'm talking about software tools that speed up my work, not specific tools for ruby or rails (but I can talk about some of those in a later post. Let me know what you want to read about down below in the comments). Before I get started, let me emphasize something:

These tools are not magic

What do I mean by that? Well, I mean that if you start using one of these tools tomorrow, your development speed is not going to immediately double. In fact, it will probably slow down. These tools take time to learn and master. I certainly don't consider myself to be a master of any of them. However, I have used them, along with other tools, for quite a while now, and I find that since I've learned them and use them often, I am much better off than I would be if I hadn't ever learned to use them.

With that out of the way, let's jump in. Here are some tools I can't live without:

Linux

I'm a Linux guy. Well, I'm also a Mac guy, but as far as development is concerned, Linux is the way to go in my opinion. Specifically, I use Ubuntu because of its ease of use and popularity (and thus wide community and support base). It has lots of great programs bundled in, is easy to configure and do all that lovely sysadmin stuff with, and typically "just works". There are so many great things about Linux and Ubuntu that I can't get into here, but suffice it to say that development would be a much more difficult and cumbersome task for me without them.

For more information, check out http://www.ubuntu.com/ and http://www.linux.org/

Zsh

With Linux comes the command line, or shell. I used the default Ubuntu shell, bash, for a long time. Relatively recently, I've switched over to zsh. I like it better for a few reasons, including the prompt customization, the superior autocomplete functionality, and powerful shell globbing. Not that I've gone really deep into any of these, but it certainly has made my life easier since I've switched. I also use "Oh My Zsh" for configuration, which is a very convenient way to get a great default configuration setup.

Even though I use zsh instead of bash for a few reasons, really the tool I can't live without is the Linux command line. Bash, zsh, whatever. After having learned to use the command line, I can work faster, get tasks done easier, and of course, write scripts to do the things that I don't want to do by hand.

To read up on zsh, head over to http://www.zsh.org/ and swing by https://github.com/robbyrussell/oh-my-zsh to check out a great configuration framework.

Vim

Ah, vim. What can I say? Every developer needs a good text editor, and vim is one of the best ones out there. Very customizable, loads of plugins available, pretty steep learning curve but so very worth the effort.

I guess the big benefit of vim (or any good text editor for that matter) is that it makes it easy and fast to do the things I need to do often. This is so important. As a software developer, my job is to manipulate text. One way to do that, of course, is on the command line, using scripting and such things. However, my primary tool for manipulating text is my text editor. I need my text editor to do what I want it to do, when I want it to do it, and quickly. The commands should be intuitive (or at least easy to remember), and things that I do often should be very fast for me to remember and execute. Also, it should look nice (syntax highlighting for code, for example) and give me specialized functionality without too much hassle. I've found vim to meet all of these expectations for me.

Head over to http://www.vim.org/ to learn more about this great editor.

Git

Ok, so I believe that everything should be under version control. When I was in school, I wrote my assignments and even my thesis in version control. My resume is in version control. And, of course, all of the code that I write is in version control.

Git is a very powerful distributed version control system. It is written by Linus Torvalds, who also created Linux, and it has a great feature set and a large community of users. There are many different version control systems out there, and a lot of them are very good. I personally like a lot of things about how git works (local repository, nice workflow with adding and committing, excellent features such as cherry-pick and rebase, etc.). I haven't used very many version control systems, but git certainly hits the spot for me.

Another very attractive aspect of using git is Github. Github is a web service which is excellent for sharing and collaborating on development projects with others. It makes it very easy to check out new projects, and even fork them and create your own version. Many open source projects, especially in the ruby on rails community, are hosted on Github.

Take a look at http://git-scm.com/ and https://github.com/ to learn more.

Tmux

Ever found yourself in a terminal emulator with way too many tabs to keep track of? Me too. One day I got sick of it and started using "screen". Shortly thereafter, I switched to tmux, a newer and better maintained alternative. In short, this allows me to have several "sessions" with several "windows" of terminals. Each session and window has a name which allows me to keep them all organized. It's easy to switch between them, create new ones, etc. I can even attach to a single window from multiple locations at once, allowing me to ssh into my work machine from home and pick up precisely where I left off in the exact same terminal instance.

It's a little hard to explain without going on forever, but it basically lets you organize several terminal instances and switch between them easily, much more easily than if you use tabs. If you're using tmux, or considering using tmux, I would highly recommend checking out tmux_resurrect (https://github.com/alexsanford/Tmux-backup). It lets me save and reload my tmux sessions when I need to reboot my machine, which is a huge productivity boost.

Read more about tmux at http://tmux.sourceforge.net/

Chrome (or Firebug)

And finally, what is a good web developer without a good browser? I use chrome, personally, but also use all of the other browsers from time to time. One thing I like about chrome (Firefox's "firebug" plugin is very similar) is the web debugging tools. It has a full Javascript debugger, DOM tree inspector, allows me to change CSS rules on the fly, etc. Very useful both for learning how to get something to work, and for debugging issues.

Well, there you have it. These are some tools that I use every day, have improved my workflow, and I highly recommend that you try out. I haven't even gotten into any of the specific ruby and rails tools that I use such as rvm, but that will have to be the topic of another post. For now, good luck trying out some new tools! Oh and by the way, you can check out my configuration for some of these at https://github.com/alexsanford/config if you're interested.

What do you think? Any tools that you use everyday that you would like to mention? How about one of the tools in my list that you think you would like to try out? Leave a comment below.

Vim Motion Commands

Hello again. Time for my first real "tech" post. For my first post, I feel that I should write about the one tool that I use for every software project, every programming language, and even writing notes and documents. Which tool is that? Well, as implied by the title, this tool is my favorite text editor. Vim.

DISCLAIMER: I am not going to participate here in the "holy wars" between all the text editors out there. I have not tried many alternatives, although I have tried a few. There are many really great text editors out that, and I encourage you to try a few. I happen to use vim because I like it. End of story.

For this post, I would like to discuss vim motion commands. These simple shortcuts are key to dramatically increasing my text editing speed. Let me give you a little run-down of what I'm talking about.

At the most basic level, motion commands are just simply keyboard shortcuts which can be used to navigate around in a file. If you've used vim at all, you'll know about the basic left/right/up/down commands, respectively, the keys 'h', 'l', 'k', and 'j'. These basic motions are supplemented by a great deal of very useful commands which allow you to move around much more quickly. Here are a few examples which I use daily:

• w - move to the beginning of the next word
• W - move to the character just after the next space (subtly different from w)
• b - move to the beginning of the previous word
• B - move to the character just after the previous space (subtly different from b)
• % - move to the matching character. I.e. if I am currently on a '(' character, move to the matching ')'
• t<character> - move to the character just before the next `<character>` (i.e. the command 't,' would move to just before the next comma)
• f<character> - similar to the above command, but move the cursor to `<character>`, rather than the character just before it

And there are many more that are useful as well.

Now the real point of this post is this: these characters are not only useful for moving around in a file. Most of the time, I'm using these motions in conjunction with other commands. For example, if I want to delete everything up to the next comma, I will simply type 'dt,'. Broken down, this means "delete to ,". Simple, eh? Now, if I want to delete everything up to and including the comma, I will instead type 'df,'. This means "delete find ,". Doesn't make quite as much sense when you say it out loud, but you get the picture.

Finally, there are certain motion commands that only really make sense in the context of performing some command on the text, but they are extremely helpful. Let me give you an example.

Say I want to replace all the text inside of the current parentheses (i.e. my cursor is between a pair of parentheses). I could go back to the opening parenthesis, type 'ct)' (change to ) ) and start typing right? Well, yeah that would work. But another way to do it is to type 'ci('. This means "change inner (". This will remove everything inside the parentheses and place your cursor in between them in insert mode. Give it a try. It's pretty neat.

Of course, there are many variants of this. Some examples to give you a taste:

• ciw - change inner word
• da( - delete outer (  --  no, that's not a typo. the letter is an a, but I sorta pretend in my head that it stands for outer. Hey, whatever works, right?
• vi{ - visual select inner {

And so on.

So there are a few thoughts on vim motion commands. Sorry if this first post has ended up being a little lengthy. Still getting use to the whole blogging thing. But let me stress this: learning to use vim motion commands will dramatically increase your workflow. No more holding down the arrow keys to move over to where you want to go. No more wasting time fussing with the mouse trying to select exactly what you want, accidentally deleting the opening parenthesis by mistake, having to type it back in, etc. Learning vim motion commands is definitely worth taking the time to do.

How about you? Any vim motion commands or other neat tricks that speed up your workflow? Leave a comment below with your thoughts (assuming that by the time you read this I've figured out how to enable comments...)

Welcome to the blog!

Hi there,

Well, I've been thinking about starting up a tech blog for some time now, and tonight I finally decided to stop thinking about it and start doing it. So here we are. Real glad you came by to join me.

As you probably have gathered, I'm a software developer by trade. I especially enjoy web-based software and my preferred platform is Ruby on Rails. Beyond that, however, one thing I love about my job is that I learn something new every day. Literally. I love discovering new and better ways to do things, overcoming challenges, and learning how to use cutting-edge technology to make my life easier. In this blog, I want to share some tips and tricks with you that I'm learning along the way. I hope that this will be as fun for you to read as it is for me to write, and I hope the random little tidbits that I learn will help you out.

So that being said, I'm gonna stop blabbing on here and start getting into the fun stuff. See you on the next post!