TL;DR - By exploiting Webpack’s async loading feature you can roll a feature-gating mechanism in your JavaScript app that only loads the code your end user is supposed to see. This is great for beta testing features and making smaller initial loads. You probably don’t need to do this in your application.
For the past year I’ve been working with an extraordinary team building an administrative interface for The Groundwork, a SaaS technology platform built to help nonprofits collect supporter data, raise funds, host events, measure progress, and more. This has been extremely fulfilling work, primarily because our platform is purpose built to help organizations like the UNHCR and similar non-profits/NGOs. Aside from the mission and clients, building Admin (yes, that’s what we call the administrative application - catchy isn’t it?) has been a treat as it’s presented us with a number of tasty challenges.
Oh, by the way: You can read an excellent design overview about Admin from my developer-in-arms Nikki Lee.
When planning the initial architecture for Admin I was presented with a few challenges right off the bat:
After some time noodling through the problems and hacking on some ideas, I settled on the front-end stack du jour: React / React Router / Redux / Webpack / Babel (or The RRRWB Stack as I never, ever, call it). These libraries provided a lot of tools and features which helped me deal with the project requirements and provided numerous conveniences, such as the abilities outlined below.
Webpack provided an easy win for the first requirement with its ability to break and application into multiple JavaScript files. The CommonChunkPlugin
let us divide our application and library code into separate compiled files (app.js
and vendor.js
respectively). vendor.js
is a fairly hefty file, but it contains all of the library code we need for the app and is only loaded once at the beginning. app.js
is a smallish file which contains the bare minimum of application code needed for the initial page load, again only loaded once.
The other big win was Webpack’s concept of split points which allowed us to subdivide the application code further. Using require.ensure
to pull in modules (instead of ES6’s import
or the regular variant of CommonJS require
) tells Webpack to split the app code into individual numbered files, (ex: 13.js
) that are requested via Ajax and loaded into memory. What this essentially gave us was a “load on demand” architecture for the rest of the application past the initial load of app.js
and vendor.js
. We were able to achieve this through a nice trick tucked away in React-Router: dynamic asynchronous routes. You can see an example of this within their huge-apps demo. This is an example of one of our routes:
:::javascript
export default {
path: 'events',
documentTitle: 'Manage Events',
getChildRoutes(location, cb) {
require.ensure([], (require) => {
cb(null, [
require('./routes/CreateEvent'),
require('./routes/Event')
]);
});
},
getComponents(location, cb) {
require.ensure([], (require) => {
cb(null, require('./components/ManageEvents'));
});
}
};
When you navigate to /events
in Admin, an XMLHttpRequest
is made for the correct component file and loaded into memory. The system is smart enough to only load them once. Webpack provides this essentially for free (after sweating through the elaborate configuration details).
This does add some complexity to your application, in that you cannot define your Routes all up front in JSX as one normally does with React-Router (example). We adopted a file structure similar to the huge-apps
example that bundled the route definitions within the index.js
file into folders with their components and child routes:
:::bash
src
├── routes
│ ├── ManageEvents
│ │ └── 0.0.1
│ │ ├── __test__
│ │ ├── actions
│ │ ├── components
│ │ ├── config.js
│ │ ├── index.js
│ │ ├── reducers
│ │ ├── routes
│ │ ├── selectors
│ │ └── utils
This helped solve two of my bullet points: Load the app on demand to keep initial file size down and also break the app into discrete routes, or higher-level components, which developers could work on independently without stepping on each others toes too much. The next big problem to tackle was feature-gating the application.
Feature gating boils down to: use some criteria to present different code paths to different users. In its simplistic form this could be something like:
:::javascript
import Bar from 'bar';
import Default from 'default';
import Foo from 'foo';
import User from 'user';
switch (user.getFeature('events')) {
case 'FOO':
render(Foo);
break;
case 'BAR':
render(Bar);
break;
default:
render(Default);
break;
}
There are lots of ways to approach this, and you can use feature gating for many different things: beta testing new code, restricting access, deploying custom features for specific users, etc. There are few things to consider when setting up a system like this:
In our case we not only wanted to change code paths on a user by user basis, we wanted to do it a level up, on an organization by organization (org) basis (an individual user in our system always belongs to an organization, such as UNHCR). We also wanted to completely switch off parts of the application at an org level.
Luckily, as we had already implemented async code loading based on routes, we already had an entry point. By controlling the routes given to React Router we could control what code was loaded into the application, sparing a user the kilobytes for modules they didn’t need nor had access too. To pull this off we leveraged some interesting features of our platform.
The Groundwork is built on a micro-service architecture. A big part of running micro-services is configuration management, ensuring all the individual services know where to find each other and configure themselves properly on startup. Early on we leveraged Consul to help with configuration and service discovery, using it as is and building additional services on top of it. One of those additional services was a system which allowed us to store information about each organization that uses our system. We could use this system to store feature metadata for each org, requesting it on demand to build the routing tables for React Router when the application bootstrapped on page load.
To pull this off we created two different sets of metadata: Feature Definitions, which were stored in Consul and specified what components an org has access to, and a Component Catalog compiled into the application which provided a directory of all the available components within the app (but not the component code itself.)
Example feature definitions (JSON - stored in Consul on a per organization basis):
:::json
[
{
"AccountWorkspace": {
"version": "1.0.0",
"children": [
{
"accountChangePassword": {
"version": "1.0.0"
}
}
]
}
},
{
"OrgSettingsWorkspace": {
"version": "1.0.0",
"children": [
{
"orgSettings": {
"version": "1.0.0"
}
},
{
"orgSettingsAdmins": {
"version": "1.0.0"
}
},
{
"orgSettingsBilling": {
"version": "1.0.0"
}
}
]
}
}
]
Example component catalog (JavaScript - compiled into app.js
):
:::javascript
const catalog = [
{
feature: 'AccountWorkspace',
version: '1.0.0',
path: '/account',
route: {
path: '/org/:org/account',
component: require('./containers/AccountWorkspace')
}
},
{
feature: 'OrgSettingsWorkspace',
version: '1.0.0',
path: '/organization-settings',
route: {
path: '/org/:org/organization-settings',
indexRoute: require('./routes/OrgSettings/1.0.0'),
component: require('./containers/OrgSettingsWorkspace')
}
}
// ...
]
When the application bootstrapped on initial page load it would take the Feature Definitions provided by Consul and combine them with the Component Catalog to produce two new objects: our Global Navigation tree and a React Router routing table. This means that the Feature Definitions from Consul determine how we render the navigation of the entire application (including in what order items appear in the nav system) as well as the available routes. If you don’t have access to a component, a link to it is never rendered in the application.
In addition to controlling access and order, we implemented a versioning scheme which would allow us to specify which flavor of a component to serve to particular orgs:
:::bash
└── ConfigureIntegrations
├── 1.0.0
├── 1.1.0
├── 1.2.0
├── 1.3.0
└── 1.4.0
Our feature definitions for an org could specify which version of a component to load from the catalog. For instance, as the Platform team rolled out new features or API changes, we would create new versions of a component taking them into account. We could then deploy these changes to our dev environment and update our feature definitions in Consul to see the new verions. Additionally this provided a fail-safe in prod, as we could deploy the new versions and then only give ourselves access in the feature definitions for a final sanity check against prod data. We could then switch on the feature definitions for all orgs (or a subset) once we were happy.
This system definitely added some complexity to the application, but also provided a lot of power and flexibilty in how we developed and deployed code for the end users.
Top image: Pies, 1961 by Wayne Thiebaud
]]>A while back I put together a little graphics package written in Racket for generating identicons. This summer someone at work stumbled across it and asked if I could generate some of those identicons in the browser. I decided to give it a shot and whipped up a small subset of the original package in ClojureScript: identikon-cljs
You can see the compiled JavaScript in action at https://pool-chatter.gomix.me/
To get an interactive development environment run:
:::bash
lein figwheel
and open your browser at localhost:3449. This will auto compile and send all changes to the browser without the need to reload. After the compilation process is complete, you will get a Browser Connected REPL. An easy way to try it is:
:::clojure
(require '[identikon-cljs.core :as identikon] :reload)
(identikon/make-identikon "#idk0" 200 200 "identikon")
and you should see an identikon appear in the browser window:
From JavaScript you can import /resources/public/js/compiled/identikon_cljs.js
and then use it like so:
:::javascript
identikon_cljs.core.make_identikon("#idk0", 300, 300, "identikon");
identikon_cljs.core.make_identikon("p.idkp", 60, 60, "small identikons");
Since the original package was written in Racket it made a lot of sense to use a Lisp for the browser version as well. The only real player in this space at the moment is ClojureScript, which I’ve used before on other projects. The downside to ClojureScript is it brings along a largish compiled runtime and I wanted to keep this as small as I could. The Google Closure compiler does a great job with dead code elimination, but the runtime still incurs a hit, so I decided to do away with SVG rendering libs and generate elements on my own using Hiccups. I also needed to generate SHA-1 hashes of input strings, so I leaned on cljs-hash. Finally I needed to manipulate HSB and Hex colors so I included the excellent color library.
All in all it was fun to port Racket over to ClojureScript and also re-think a few things in the process.
]]>This weekend I decided to finally bite the bullet and deploy one of my Clojure apps as a JAR. “Certainly” I thought to myself, “there are great tutorials for this online”. Yes, yes there are, except none of them worked for me and in fact did lead to the consumption of some rye whiskey.
Caveats: Every Clojure app is different, and the details of your app will determine the steps necessary to build a JAR. None of the information I found online specifically addressed the problems I had, so I made this post for those who have a similar application with similar needs. YMMV.
This post will walk through distilling a small Compojure web-app (codegumi.com) into a nice JAR for deployment on a VPS. This is an older app that gathers images from the Flickr API and dumps them into your browser. I used it to teach myself ClojureScript and core.async back before Om and React hit the scene.
The application needs to compile ClojureScript and has static resources. It also works with the VPS’s file system (caching searches as JSON files), a small feature that caused much pain.
We still want the project to work in development with a simple lein ring server
as well as a standalone JAR file. You can read all the source code if you like.
Prior to this exercise I deployed my app the way real hackers deploy apps, in the most ridiculous way possible: I tmuxed into my VPS, did a quick lein trampoline ring server-headless 8080
and detached. This worked great! Except when my VPS got restarted or crashed (as they do) and I would then have to manually shell back in and restart the app.
Creating a JAR would allow me to create an upstart script so the OS could restart my app when necessary. It also seemed like the right thing to do.
Previously I attempted to install Tomcat and deploy a WAR file, all of which failed miserably. The concept of a JAR file with an embedded server is very attractive to me and seems like something I should know how to do, so onwards!
You want to make a JAR, so you quickly drop “clojure jar” into DuckDuckGo and you will quickly see something about: lein uberjar
. You type this into your terminal, and the whole world explodes into a Java stacktrace. “But, but, my app ran fine with lein ring server
!” you despair. Yes, yes it probably did, however JARs are different animals. Fickle animals with sharp teeth. Lets take a look at how they work.
To make a JAR you need to have a -main
function in your app. In a lot of Compojure tutorials you just define a ring
handler in your project.clj
:
:::clojure
(defproject codegumi "0.2.0"
:description "codegumi.com in Clojure/ClojureScript"
:url "http://codegumi.com"
:license {:name "Eclipse Public License"
:url "http://www.eclipse.org/legal/epl-v10.html"}
:dependencies [[org.clojure/clojure "1.5.1"]
...
[ring "1.3.2"]]
:plugins [[lein-cljsbuild "1.0.1-SNAPSHOT"]
[lein-ring "0.8.8"]
[lein-environ "0.4.0"]]
:source-paths ["src/clj" "src/cljs"]
:ring {:handler codegumi.handler/app} ;; <---- Ring Handler
When you run it with lein
the handler is picked up and passed to ring
from your Clojure file and away you go. This won’t work with a JAR, you need to define a main entry point for your application. It can be as simple as passing your handler to jetty:
:::clojure
(ns foo.handler
(:gen-class) ;; <--- you need this too!
(:require [compojure.core :refer :all]
[compojure.handler :as handler]
[compojure.route :as route]
[ring.adapter.jetty :refer :all] ;; <-- need a server!
...
))
;; Compojure routes
(defroutes app-routes
(GET "/" ... ))
;; Ring handler
(def app
(handler/site app-routes))
;; Pass the handler to Jetty on port 8080
(defn -main []
(run-jetty app {:port 8080}))
So in the above we did a few things:
(:gen-class)
which Uberjar needs.ring.adapter.jetty
to embed the Jetty web server in our JAR.-main
function which passes the ring
handler to jetty.Did you catch that part about embedding jetty into the JAR? That’s right, our JAR will contain its own server. Previously ring
would spin one up for us when we did lein ring server
. Our new -main
function will handle that for us.
We need to tell Uberjar where the main file is, so we need to add a setting to our project.clj
:
:::clojure
(defproject codegumi "0.2.0"
:description "codegumi.com in Clojure/ClojureScript"
:url "http://codegumi.com"
:license {:name "Eclipse Public License"
:url "http://www.eclipse.org/legal/epl-v10.html"}
:dependencies [[org.clojure/clojure "1.5.1"]
...
[ring "1.3.2"]]
:plugins [[lein-cljsbuild "1.0.1-SNAPSHOT"]
[lein-ring "0.8.8"]
[lein-environ "0.4.0"]]
:source-paths ["src/clj" "src/cljs"]
:main codegumi.handler ;; <---- Where -main lives
:ring {:handler codegumi.handler/app}
If your app is very simple, these may be the only changes you need to build a JAR. You should give it a shot and see what happens. In my case I needed to do more research to deal with files and compiling ClojureScript.
When I first tried to build a JAR, after creating my -main
function, I got tons of errors about accessing directories that didn’t exist within the JAR’s classpath. Oh noes. The one thing every non-Java person fears is any mention of classpath, as this word is burdened with much lore and spoken through the gnashing of teeth on many a message board.
I got these errors because my app needs to read/write JSON files to function and I was doing it all horribly wrong when it comes to JARs. So lets go over some basics.
Your typical Clojure/Compojure app has a directory structure something like:
:::bash
├── LICENSE
├── README.md
├── doc
├── project.clj
├── resources
├── src
├── target
└── test
In my case I have clj
files for the server side and cljs
files for the client in src
:
:::bash
src
├── clj
└── cljs
My Compojure app has a number of static resources, like css, images, etc. I also need a place for my compiled ClojureScript to live. The convention is to drop those into the resources/public
directory, as that will be included in the JAR automatically:
:::bash
resources
└── public
├── css
├── favicon.ico
├── fonts
└── img
If you’re like me you don’t come from a Java background and you don’t know much about JAR files. By default anything you put in resources
will be packed into the JAR you compile and made available to your Clojure code. However, it isn’t accessible as a filepath because it is on the classpath. Within your app you access these files in a couple of ways.
As mentioned above, Compojure defaults to looking for classpath files in resources/public
. It also gives you a route/resources
method to handle this:
:::clojure
(defroutes app-routes
(GET "/" [] ... )
(route/resources "/")
(route/not-found "Not Found"))
All of my css, font and image files are now available from the root within my app. When it’s running my template can call /css/styles.css
and Compojure will pull that file from the classpath and serve it up.
If you needed to access a file from the resource
directory from within your Clojure code, you would use (clojure.java.io/resource "filename")
. This would return a classpath which you could use to read the file.
Now, these files are effectively static. JAR files are basically zip archives. You can read files within them, but you writing to them is probably not a good idea. It would mean mutating the JAR at runtime. This is important, because my app needs to write to the filesystem to cache JSON responses from Flickr, so it needs to know where to park those files and find them later.
You may say to yourself, “I will define the path to my directory in the app!” and do something like (def json "foo/bar")
. That may work great when running locally with lein
, and it might work in a JAR depending on what you put in there. It might not, and it might not because a JAR is run by the Java Virtual Machine (JVM) and the JVM may have a different idea about relative paths than you do. You also may not want to embed a directory path into your application.
There are different ways to approach this problem. You might make the writable path an environment variable. You might, like I did, make it an argument that you pass into the JAR when you execute it. You could do both. Either way, you should determine how to tell your app where to find/write files, and use clojure.java.io
to access them. I ended up packing the filepath passed in as an arg into an atom that I could use elsewhere in the application.
To recap: Use (clojure.java.io/resource "foo.css")
to access files within your JAR’s classpath and (clojure.java.io/file "foo.json")
to access a filepath outside the JAR.
Now, assuming your project has ClojureScript, you have probably already been compiling it with lein-cljsbuild
. You are probably building it right into your resources/public/js
folder via your compiler settings. This works great when your running with lein
. This works not so great when you need to build a JAR.
Luckily fixing this isn’t a big deal. As we mentioned above, we want our ClojureScript to compile into resources/public/js
so it will get packed into the JAR with our other resources files. All we need to do is add some hooks to our project.clj
file to make this happen during an Uberjar build:
:::clojure
(defproject codegumi "0.2.0"
:description "codegumi.com in Clojure/ClojureScript"
:url "http://codegumi.com"
:license {:name "Eclipse Public License"
:url "http://www.eclipse.org/legal/epl-v10.html"}
:dependencies [[org.clojure/clojure "1.5.1"]
...
[ring "1.3.2"]]
:plugins [[lein-cljsbuild "1.0.1-SNAPSHOT"]
[lein-ring "0.8.8"]
[lein-environ "0.4.0"]]
:source-paths ["src/clj" "src/cljs"]
:hooks [leiningen.cljsbuild] ;; <--- Hook for uberjar to grab
:main codegumi.handler
:ring {:handler codegumi.handler/app}
:profiles
;; Uberjar profiles entry to specify how CLJS is compiled
{:uberjar {:aot :all
:cljsbuild {:builds [{:source-paths ["src/cljs"]
:compiler {:output-to "resources/public/js/script.js"
:optimizations :simple
:pretty-print false}
}]}}})
Adding :hooks [leiningen.cljsbuild]
will tell Uberjar to follow any CLJS compile directives when building the JAR. The compiled files, since they are in the resources
folder, will get picked up.
The addition of the :uberjar
entry in :profiles
give you some control over how your ClojureScript is compiled for JARs, which are usually meant for production.
You also may have noticed the addition of the {:aot :all}
directive to :uberjar
. This tells Leiningen to Ahead of Time compile your Clojure source into JVM bytecode as part of the JAR building process. There are pros and cons to this, so if you run into issues you might remove this directive and do some more reading.
So after much trial and error you may have generated a JAR. Cool. By default Uberjar builds them in /target
. We’re interested in the standalone JAR as that is self-contained and has everything we need. Lets go ahead and run it: $ java -jar target/my-cool-app-standalone.jar
.
If the stars are aligned and you have sacrificed the appropriate items to your gods of choice there might be a working application on whatever port it was told to run on. Say, http://127.0.0.1:8080.
It may have exploded into a Java Stack Trace in which case remain calm, pour two fingers of rye whiskey into a glass and settle in for some debugging time.
Once it is working, go ahead and upload it to your VPS and see if it runs there as well. Make sure you have Java installed and all the directories your app needs to run, environment vars, etc. If you do, it runs and you’re on Ubuntu we can now make a simple Upstart script so the OS can launch it.
Writing upstart scripts is a topic unto itself. I’ll give you a quick starter one, but you may need to do some research to meet your needs: The Upstart Cookbook.
Shell into your VPS and cd
into /etc/init/
. You will need sudo
powers to make this file, and name it after your app, something like coolapp.conf
:
:::bash
# Start the process when the VPS comes up, and stop it when it goes down
start on runlevel [2345]
stop on runlevel [016]
# If the process crashes, restart it up to 10 times in 5min
respawn
respawn limit 10 5
# Log stdout to file
console log
# Set these to the desired uid/gid you want Java to run under
setuid clojure
setgid clojure
# Set any environment vars you might need
env BAR=quux
chdir /path/to/jar
exec /usr/bin/java -jar cool-app-standalone.jar
Save that file, and then run it $ sudo start coolapp
. If all goes according to plan, your JAR should get launched and you should see it running with ps
or top
. It should be running as whatever user/group you instructed it to. Point your browser at it and see how it goes. If it does huzzah!, if not, pour some more whisky into your glass.
Some things to note: your Upstart script is (in this case) run by the system, so it won’t have access to ENV
vars you set in your user configs, this is why we set them in the conf
script.
Upstart will automatically log output to /var/log/upstart/coolapp.log
. This will help you when things go wrong (and they will.) You may also consider using alternative logging libraries available to Java/Clojure.
I’d like to give a big shout out to my man Aaron Bull Schaefer who cleaned up my initial shot at an upstart script. If you need to know anything about running infrastructure or Factor he’s your man.
Hopefully you found this collection of information helpful in packaging up your application. Maybe you just got loaded. Either way, I hope you had fun.
Helpful references
Photo courtesy of Eli Christman via Flickr - Creative Commons
]]>I’ve always liked identicons, which WordPress and GitHub have used to great effect. The premise is simple: take a user identifier such as an IP or email address and deterministically convert it into an image based on a simple algorithm. To that I end I started hacking on Identikon - a little Racket program that generates different types of identicons based on rules modules.
I like Lisps and Schemes and have been experimenting with Clojure for a while as well as noddling around with The Little Schemer and SICP. More importantly though, Racket has a great programming environment with lots of batteries such as the 2htdp/image library for working with graphics. The language was designed as a teaching tool and comes with great documentation.
One of the biggest selling points for me was Matthew Butterick’s Pollen, a Racket type-setting application which he used to publish Practical Typography. He gave a very inspirational talk (video) at RacketCon 2014 which you should check out. Pollen speaks to my art/design background so I decided to take Racket out for a test drive. I’m still very much a noob, and plan to update Identikon as I gain more experience with the language.
I wanted Identikon to do a few things:
Let’s look at each of those item in a little more depth.
I had a lot of fun exploring different systems for generating identicons, with varying degrees of success. Some examples:
Squares | Circles | Q*bert | Nineblock |
---|---|---|---|
:::clojure
(identikon 300 300 "racket" "squares")
(identikon 300 300 "racket" "circles")
(identikon 300 300 "racket" "qbert")
(identikon 300 300 "racket" "nineblock")
Each of these images is an identicon for the string “racket” using a different set of rules for generating the image. Racket organizes code into modules which expose a set of public functions to the code that requires it. To make it easy to generate rules for identicons I wanted to keep things simple, so each rules module only has to provide a draw-rules
function with the following signature:
:::clojure
(draw-rules width height user)
In return, draw-rules
has to return an image at the requested size, which is then written to a file or displayed in the REPL depending on how you’re using Identikon. The implementation of draw-rules
is completely the business of the rule module and hidden from the rest of the program.
width
and height
are the pixel dimensions of the image you’re generating, and user
is the data representation of the string given to identikon
. To make unique images we need unique data, so the user
string is first converted to a SHA-1 hash, and then into a list of 20 base-10 numbers, so that “racket” is converted into:
:::clojure
(process-user "racket")
; returns
'(27 180 200 176 189 77 68 156 1 211 209 117 218 72 146 38 144 184 241 76)
This gives the rules module a consistent set of data to work with in generating a unique image. You can use these numbers in a variety of ways, such as generating RGB colors and such. One of the strategies I used in the circles and squares rules is to take the first 15 numbers from the list and break that into 5 lists of 5 mirrored numbers:
:::clojure
; user is '(27 180 200 176 189 77 68 ... 76)
(chunk-mirror2 (drop user 5) 3)
; returns
'((27 180 200 180 27)
(176 189 77 189 176)
(68 156 1 156 68)
(211 209 117 209 211)
(218 72 146 72 218))
Now we have a 5x5 grid of values to fold over, turning pixels on/off depending on the value of the items (ex: even, odd, etc.). Since the list values are “mirrored” the generated image will be symmetrical. I then use the 5 values I dropped from the initial user
list to generate a range of color values to use for the circles or however I see fit. You’re pretty much left to your imagination as to how you use the values generated from the initial string, as long as the output is deterministic.
Other systems are much more complex, like nineblock which packs drawing functions into a vector to be accessed numerically as needed. You can read about nine blocks here.
Of course, once you have something working in a REPL, you can start to have a lot of fun playing around:
:::clojure
(for ([n (range 400 496)])
(print (identikon 80 80 (number->string n) "squares")))
Another feature I wanted was the ability to create new rules modules on the fly in discrete files, and allow identikon
to pull them in without any prior registration or configuration. As long as a module followed the draw-rules
API it should be usable. This posed some challenges and required digging into Racket a bit deeper than I planned.
Lisps and Schemes (and therefore Racket) treat code as data, which is a big concept I’m not going to get into deeply here. Suffice it to say, a running Racket program can load in source code from a file, manipulate it and run it within the current environment. Racket manages environments with namespaces - which manage all of the mappings of identifiers to bindings and module instances.
To solve my arbitrary rules module function I needed to create a function that would dynamically load a Racket module file into a new namespace, and then shoehorn that into the existing namespace. Yes, that sounds a little nuts, so lets walk through it (from identikon.rkt):
:::clojure
; Define the draw-rules function as a null value, we will overwrite
; this with the dynamically loaded module's version of draw-rules
(define draw-rules null)
; Create a handle for our new namespace
(define-namespace-anchor a)
; Where we keep the rules modules
(define RULES-DIR "rules")
; Dynamically load in a rules file, attaching it to a new namespace
; that has access to the 2htdp/image module functions (for drawing)
; and make it available to this namespace with (dynamic-require)
(define (load-plug-in file)
(let ([ns (make-base-empty-namespace)]
[filename (build-path (current-directory) RULES-DIR file)])
(namespace-attach-module (namespace-anchor->empty-namespace a)
'2htdp/image
ns)
(parameterize ([current-namespace ns])
(dynamic-require filename 'draw-rules))))
; ... later down inside the identikon function ...
; Overwrite the draw-rules definition with the draw-rules provided
; from the rules file in load-plug-in
(set! draw-rules (load-plug-in rule-file))
Now when we make a call like (identikon 300 300 "racket" "circle")
the program will load the rules/circle.rkt
module and replace the default draw-rules
identifier with the function provided by that module.
This is pretty cool, we can hot-swap the draw-rules
function with a new one whenever we need to. However, this does negatively effect performance, as we have to hit the filesystem for the rules module every time we make an identicon. If I was going to use this to generate identicons at scale I would directly require
just the module I needed and compile it into the program. As a future project I would like to allow a user to compile a version of Identikon with a specific rules module baked in, perhaps via code re-writing during compilation with Racket’s raco tool.
Identikon will generate images within a Racket REPL simply by calling the identikon
function with valid parameters. If you pass it a file extension it will save the identicon to your filesystem.
You can do the same thing from the command line by running identikon.rkt
with racket and passing in arguments:
:::bash
$ racket identikon.rkt -s 300 -n racket
This will save out a file called racket.png
containing a 300 pixel default identical. You can see the available arguments with -h
.
Additionally, you can include Identikon into another Racket program:
:::clojure
(require "identikon.rkt")
(define foo (identikon 300 300 "racket"))
We achieve this flexibility very easily in Racket by using submodules. Within identikon.rkt
we define a special main
submodule that is only executed when you run the program from the REPL or command line. This is where we place our command line handling code:
:::clojure
(module+ main
(require racket/cmdline
racket/list)
... handle command line options ... )
This submodule won’t be executed when identikon.rkt
is required into another module. Submodules allows us to interleave code within a program using different environments, which can be pretty powerful. You can use this same technique to interleave your tests within the source code:
:::clojure
(module+ test
(require quickcheck
sugar)
; rhombus-offset calculcation is correct
(define rhombus-offset-outputs-agree
(property ([num arbitrary-natural])
(let* ([onum (rhombus-offset num)]
[diff (- num onum)])
(= num
(* diff 4)))))
(quickcheck rhombus-offset-outputs-agree))
The tests will only be run in a REPL or when you run $ raco test filename
. In this instance I’m using a Racket port of Quickcheck to do some generative testing on one of my functions. You can see a lot more of these at work in the utils.rkt file. I’ve uncovered a lot of loose bugs using quickcheck.
In general I’ve been very happy with Racket for this project. The documentation is good, and there is a mailing list and IRC channel stocked with helpful folks. If you use DuckDuckGo you can quickly search the docs using the !racket
shortcut. Additionally there is a great book, The Realm of Racket, which teaches the basics through game programming. I’m still working through it, but its helped me immensely.
The language itself is big. Unlike its forbear Scheme, it contains lots of sugar and special forms for recursively iterating over different data structures. This helps to reduce the amount of code you need to write.
Racket is still a little niche at the moment. While full-featured it doesn’t have the big community that a language like Ruby or Clojure has, so you have to do a bit more searching for examples and guidance. Also, since I’m coming from Clojure I have a tendency to lean heavily on let
and its recursive cousin let*
in my functions. Racket folks tend to prefer using define
within a function body. This makes my code a little shizophrenic looking at times, and is something I need to work out. Also, while Racket does have immutable data structures, its unclear if they’re persistent or how they’re implemented or compare to Clojure’s persistent data structures. Also, currently there is no RacketScript for the browser (although there is Whalesong).
That said, I’m starting to reach more and more for Racket as my go-to utility language and will definitely keep exploring it in the future.
]]>This post takes a quick look at EventedArray, a small Array-like JavaScript data structure (written in CoffeeScript) that allows you to register callbacks on accessor/mutator operations and also create fixed size buffers. All examples are in JavaScript.
The concept of Reactive Programming has been gaining a lot of traction lately, especially when working with user interfaces. To oversimplify, Reactive Programming deals with streams of data or events which can be sampled, combined and observed to try an bring order to the chaos found in your typical web application. A lot of very smart folks have written some excellent libraries to achieve this, such as RxJS and Bacon.js. Go check ‘em out.
When working on small applications I often find a need for an “observable” data structure. It would be great to have something like an Array which I could treat like a stack, with events triggered whenever values are pushed on or shifted off. I could achieve this with either of the libraries noted above, or even Backbone.js’s Events mixin, but sometimes I just need something small and simple. Also, it’s fun to learn how things work by rolling your own toy implementations.
I wanted this data structure to be as close to a regular Array as possible, with a standard Object Oriented type interface. The core library is written in CoffeeScript but you can pull it into any JavaScript project as a global or with RequireJS. Let’s take a look:
:::javascript
// Create a new data structure
var E = new EventedArray(1,2,3,4);
// It has setters & getters
E.set(5,6);
E.get(4); // returns 5, the 0 indexed value of the array
E.toString(); // "[1,2,3,4,5,6]"
// Remove values
E.remove(3);
E.toString(); // "[1,2,4,5,6]"
// Mess with the stack
E.pop();
E.toString(); // "[1,2,4,5]"
E.shift(); // 1
E.toString(); // "[2,4,5]"
// Underscore collection functions
E.each(function(i){ console.log(i*i); }); // 4 16 25
E.map(function(i){ return i^2; }); // [0,6,7]
E.filter(function(i){ return i%2 == 0; }); // [2,4]
// Raw access to the values
E.values; // [2,4,5]
So that’s cool, it behaves like an Array for the most part. Now I want to add some callbacks that are triggered when I set, get, shift, etc.
:::javascript
E.register('set', function(i){ console.log(i + ' was set on E'); });
E.set(6); // '6 was set on E'
E.register('remove', function(i){ console.log(i + ' was removed!'); });
E.remove(2); // '2 was removed!'
You can register events on most of the methods available, like
reduce
, every
, contains
and many more. Take a look at the
source
to see all the methods available.
A lot of times I want to treat my data structure like a fixed size
stack, something that will only hold n values, shifting older values
off the front as new ones are pushed onto the end. So I went ahead and
added a setBuffer
method to do just that:
:::javascript
var E = new EventedArray();
E.setBuffer(5);
E.set(1,2,3,4,5,6);
E.toString(); // "[2,3,4,5,6]"
E.set(7,8,9,10);
E.toString(); // "[6,7,8,9,10]"
So what can we do with EventedArray? It really lends itself to managing streams of values, so here’s a little DOM based animation example. Go ahead and waggle your cursor around in the box:
The code for this is pretty simple, you can view it in action
here, we have one
EventedArray with a buffer of 25 storing
Box
objects and a listener on the mousemove
event
passing those Boxes in:
:::javascript
// Create a queue to display our points
var displayQueue = $('#displayqueue');
var P = new EventedArray();
P.setBuffer(25);
// When points are set to this queue display them as a string
P.register('set', function() {
displayQueue.html(P.toString());
});
// Create a Queue that holds 25 items and attach events
var Q = new EventedArray();
Q.setBuffer(25);
// As Boxes are set tell them to appear and also set their
// [x,y] to the displayQueue
Q.register('set', function(b) {
b.showBox();
P.set(b.point)
});
// As boxes are shifted off, tell them to fade out
// and null them out
Q.register('shift', function(b) {
b.hideBox();
b = null;
});
// Drawing area
var canvas = $('#drawing');
// Mousemove listener
var onMove = function(e) {
Q.set(new Box([e.x, e.y], canvas)); // Make a Box
};
document.getElementById('drawing')
.addEventListener('mousemove', onMove);
So this nice, Boxes are pushed into the queue and shifted off, with events triggering their behavior. You could wire up more elaborate systems using this general concept, such as a simple Flickr search which filters items from one EventedArray into another:
In this case we’re using events on the filter
method from Underscore
to shift Photos off one stack and into another:
:::javascript
// Register a filter callback on stream which removes
// photos from the stream itself
stream.register('filter', function(f) {
_.each(f, function(i) { stream.remove(i) });
});
// Filter photos from Stream based on title
$filter_submit.on('click', function(e) {
e.preventDefault();
// make the search term lowercase
var filter = $.trim($filter.val()).toLowerCase();
// filter stream by the search term. This will trigger
// the filter callback on stream
var f = stream.filter(function(i) {
return i.title.toLowerCase().indexOf(filter) !== -1;
});
// Take all the filtered photos and create new elements
// in the filtered list
if (f.length > 0) {
filtered.each(function(i) { filtered.remove(i) });
_.each(f, function(i) {
filtered.set(new Photo(i.entry, $filtered, filtered_remove));
});
};
});
You could get fancy and use drag and drop events to move items from one array to another, using events to do cleanup, trigger animations and such (at that point you would probably want one of the more fleshed out libraries I mentioned up at the top).
EventedArray is a simple data structure. You could blow it out into something a bit more robust, registering multiple callbacks on a single event, feel free to expand it as a learning exercise.
TL;DR - Fogus and Jeremy Ashkenas published the underscore-contrib repo recently which is loaded with functional JavaScript goodness based on the popular underscore library. I walk through a few examples of how I use it in the hopes you’ll get interested in working out new ways to solve your own day-to-day issues.
Functional programming has been getting a lot of play recently, but many demos of languages like Clojure and Haskell can be pretty intimidating and esoteric for your average front-end developer who primarily works in JavaScript. The reality is that JavaScript packs a lot of functional power and you can exploit this in your day-to-day work to get real results writing leaner and more flexible code.
If you don’t already use or know about Jeremy Ashkenas’ Underscore.js, then just stop right now and download it. Underscore provides a rich toolset of functions that ease manipulating Arrays, Objects and Functions. More importantly it brings higher order functions like map and reduce, native to modern browsers, to older versions of Internet Explorer. (In my examples I’m going to show how to do it both natively and with Underscore.)
Michael Fogus (Fogus) put together the Underscore-Contrib library, which brings a whole new set of functional goodness to JavaScript using underscore.js as the foundation. Reginald Braithwaite1 contributes to it as well and has written two of the better books about JavaScript: CoffeeScript Ristretto and JavaScript Allongé. Fogus is working on Functional JavaScript
This post is based on a quick talk I gave at an Arc90 Code Review. The main idea is that with underscore & underscore-contrib you can think about some problems in a functional rather than imperative way, and may arrive at more flexible and powerful solutions to a given problem. Additionally I dislike programming demos that use factorial and other math concepts to explain functional programming. My own contrived examples are pulled from the day to day stuff I (and maybe you) do at work such as String and Date manipulation.
Clone the repo: $ git clone git@github.com:documentcloud / underscore-contrib.git
Or download a zip: master.zip
Demo code: functions.zip
JsFiddle: I put together a JSFiddle here which has all of the libraries loaded up and a log
function to output your work in HTML. You can use this to play along with the examples below. (The log
function itself uses underscore.js.)
Let’s say I’m working on a simple web interface for a client, and I need to do some string manipulation on data that I get from the server. I’ll need to do this repeatedly and in different ways, so it’s in my best interest to break my code out into little discrete functions that do one thing:
:::javascript
// Make the first character of a string Upper Case
var upperFirst = function (s) {
return s[0].toUpperCase() + s.slice(1);
};
// Wraps native String.toLowerCase() - will show you why in a min
var lower = function (s) {
return s.toLowerCase();
};
// Returns String prefixed by Mr.
var toMr = function (s) {
return "Mr. " + s;
};
Hey cool, I have three functions for operating on Strings, that I can use in different ways to modify my data. JavaScript functions can be passed around any which way, so hey, let’s use them!
:::javascript
var ted = toMr(upperFirst(lower('TED')));
console.log(ted) // Mr. Ted
That works, sure, but what if I need to do the Mr. transform in a lot of different places? I don’t want to type out toMr(upperFirst(lower('n')))
every time do you? So you would probably make another function like:
:::javascript
var makeMr = function(s) {
return toMr(upperFirst(lower(s)));
}
console.log(makeMr('ted')) // Mr. Ted
Okay, much better, we’re assembling functions to get things done. But hey, what if we had a function that could automatically compose other functions together for us in a generic manner? We do.
:::javascript
// underscore: _.compose()
var makeMr = _.compose(toMr, upperFirst, lower);
console.log(makeMr('TED')) // Mr. Ted
// underscore-contrib: _.pipeline()
var makeMr = _.pipeline(lower, upperFirst, toMr);
console.log(makeMr('TED')) // Mr. Ted
compose
and pipeline
are two ways to get to the same place: a new function composed of other functions. As long as each function in the chain returns a value for the next function, as all of our string functions do, you’re set. The difference between compose
and pipeline
is the order in which your functions are called. compose
calls them from right to left while pipeline
is left to right.
Now that we’re quickly composing functions, we can build up our own library for this app:
:::javascript
// Return a capitalized String prefixed by Mr.
var makeMr = _.pipeline(lower, upperFirst, toMr);
// Reverse a String
// notice the use of _.explode() from underscore-contrib to
// convert a String into an Array
var backwards = function (s) {
return _.explode(s).reverse().join('');
};
// Make a function which reverses makeMr
var backMr = _.compose(backwards, makeMr);
console.log(makeMr('TED'), backMr('TED')) // Mr. Ted deT .rM
You’re making new functions on the fly now, using simple functions like Lego to make something more complex. This is pretty powerful when you think about it: all those admonitions to make small functions that do one thing well are starting to make sense.
Let’s do something a little more tasty: imagine you want to take an array of Strings and transform them, creating a new Object using the original string as the key and the transformed string as the value. You would probably be tempted to use a for-loop, or we could do something a bit different:
:::javascript
// GOAL:
// get from ['ted','bob','jim'] to
// {"ted":"Mr. Ted","bob":"Mr. Bob","jim":"Mr. Jim"}
// Start with a tuple from makeMr() like this: ['ted', 'Mr. Ted']?
// _.juxt() from underscore-contrib returns a function that
// returns an array of the calls to each given function for some arguments.
var j = _.juxt(_.identity, makeMr);
console.log(j('ted')) // ['ted','Mr. Ted']
// Hold up! What's _.identity?
// Identity function: f(x) = x - returns its argument unchanged
// Use it when you need a function as an argument but don't want
// to do anything to it.
// So, _.juxt(_.identity, makeMr) returns
// [x, makeMr(x)] or ["ted","Mr. Ted"]
// Now let's take an Array of strings and make a new Array of tuples
var names = ['ted','bob','jim'];
// Native Array.map for modern browsers
// https://developer.mozilla.org/en-US/docs/JavaScript/Reference/Global_Objects/Array/map
var nameTuples = names.map(_.juxt(_.identity, makeMr));
// Underscore _.map if you need to be safe
var nameTuples = _.map(names, _.juxt(_.identity, makeMr));
// [["ted","Mr. Ted"],["bob","Mr. Bob"],["jim","Mr. Jim"]]
// Now make an Object from those tuples using _.object()
// which takes key value pair arrays [key, value] and
// converts them into an Object {key : value}
console.log(_.object(nameTuples));
// {"ted":"Mr. Ted","bob":"Mr. Bob","jim":"Mr. Jim"}
// Now let's do it in a terse manner without all the local vars:
var nameObj = _.object(['ted','bob','jim'].map(_.juxt(_.identity, makeMr)));
// {"ted":"Mr. Ted","bob":"Mr. Bob","jim":"Mr. Jim"}
// Iterate!
var emanObj = _.object(['ted','bob','jim'].map(_.juxt(_.identity, backMr)));
// {"ted":"deT .rM","bob":"boB .rM","jim":"miJ .rM"}
var wtfObj = _.object(['ted','bob','jim'].map(_.juxt(upperFirst, backMr)));
// {"Ted":"deT .rM","Bob":"boB .rM","Jim":"miJ .rM"}
See what we did there? By combining functions from underscore and underscore-contrib with our own functions we’re able to convert a simple Array of Strings into a new key value Object in one line of code. You can now quickly iterate through different function combinations to slightly or radically change the end Object as needed. By taking the time to reason through what you want to do you can compose a flexible solution.
There are a whole bunch of interesting function combinators to play with, way more than I can cover here.
So that was a pretty full-bore introduction into function composition and array mapping. Lets take that a little further by throwing reduce into the mix.
Another common task in JavaScript is working with Dates. I usually toss in a library like Moment.js to deal with this sort of thing, but let’s do something simple. Again, we’ll write some functions:
:::javascript
// Return a new Date based on String
var date_new = function (s) {
return new Date(s);
};
// Utility function for to_iso8601
var date_pad = function (n) {
if (n < 10) {
return 0 + n;
} else {
return n;
}
};
// Generate an ISO8601 formatted date from a Date object
// (!! there are better ways to do this)
var to_iso8601 = function (d) {
return d.getUTCFullYear() + '-' + date_pad(d.getUTCMonth() + 1) + '-' + date_pad(d.getUTCDate()) + 'T' + date_pad(d.getUTCHours()) + ':' + date_pad(d.getUTCMinutes()) + ':' + date_pad(d.getUTCSeconds()) + 'Z';
};
// Return an ISO8601 String with the time removed
var iso_no_time = function (s) {
return s.split('T')[0];
};
Cool, we can make dates and convert them to ISO8601 strings. Let’s play:
:::javascript
console.log(date_new('5/5/2013')) // "Sun May 05 2013 00:00:00 GMT-0400 (EDT)"
console.log(date_new()) // Invalid Date
Whoops! If we forget to pass an argument to date_new() we get Invalid Date
back. What if we don’t want that behavior for this application though? What if we want date_new() to have a default value? How many times have you written this kind of guard code:
:::javascript
var date_new = function (s) {
if (s === null || s === undefined) {
s = '1/1/1970';
}
return new Date(s);
};
console.log(date_new()); // "Thu Jan 01 1970 00:00:00 GMT-0500 (EST)"
You’ve probably written it a lot, and it can get tedious as times. Underscore-contrib gives us a nifty little function combinator called fnull
that makes doing this type of thing easy:
:::javascript
var date_new = function (s) {
return new Date(s);
};
date_new = _.fnull(date_new, '1/1/1970');
console.log(date_new()); // "Thu Jan 01 1970 00:00:00 GMT-0500 (EST)
fnull
checks the argument for a null value, and if true passes in the default argument you defined (‘1/1/1970’). This is where you say: “Umm okay, but it doesn’t save me that much code, and it’s not inside my function so it may not be as clear.”
You’re right, but what if your function were more like this:
:::javascript
// Lots of arguments, none can be null
var create_datetime = function(year, month, day, hour, min, sec) {
var def = [1970, 1, 1, 0, 0, 0];
if (arguments.length < def.length) {
for (var i = arguments.length; i < def.length; i++) {
arguments[i] = def[i];
}
}
return new Date(
arguments[0],
arguments[1],
arguments[2],
arguments[3],
arguments[4],
arguments[5]
);
}
// Same thing using fnull
var create_datetime = function(year, month, day, hour, min, sec) {
return new Date(year, month, day, hour, min, sec);
}
create_datetime = _.fnull(create_datetime, 1970, 1, 1, 0, 0, 0);
Not only is using fnull
less verbose, but your defaults are not hardcoded inside create_datetime
, so you can change them on the fly as needed in your application. Just another nice-to-have in contrib.
So now that we have some basic date functions, put them to work:
:::javascript
// Compose a function to make ISO8601 date strings
var make_iso = _.pipeline(date_new, to_iso8601);
// Our data
var dates = ['4/1/2012', '4/2/2012', null, '3/28/2012'];
// Turn these date strings into ISO8601 dates using map
var isos = dates.map(make_iso)
// ["2012-4-1T4:0:0Z","2012-4-2T4:0:0Z","1970-1-1T5:0:0Z","2012-3-28T4:0:0Z"]
// Let's take those isos and break the time off
var isos_no_time = isos.map(iso_no_time);
// ["2012-4-1","2012-4-2","1970-1-1","2012-3-28"]
// Notice that the null in the array was caught by
// fnull and defaulted to 1/1/1970
Now that we haves arrays full of dates, we may want to find the oldest & newest dates. Array.reduce is a great way to find those values. Array.reduce
is an accumulator, so it walks down your array left-to-right operating on the values to return a single new value (the underscore equivalent of reduce
is here). Let’s take a look:
:::javascript
// Let's find the most recent date in that array using reduce
// As the function walks down the array, it compares the current value
// to the next value on the right. If the current value (l) is
// greater than the next (r) use l as the current value in the next
// comparison, otherwise use r.
var recent = isos.reduce(function (l, r) {
return (iso_no_time(l) > iso_no_time(r)) ? l : r
});
console.log(recent); // 2012-4-2T4:0:0Z
// Let's find the oldest date in that array using reduce
// We do exactly the same as above, but we flip the comparison
// operator to l < r
var oldest = isos.reduce(function (l, r) {
return (iso_no_time(l) < iso_no_time(r)) ? l : r
});
console.log(oldest); // 1970-1-1T5:0:0Z
So, just like with our strings, we’re able to quickly compose functions to operate on dates and iterate over collections with a very small amount of code. Additionally, your intent is made clear in the code itself: “I want to reduce this array of dates to the most recent date.”
Hopefully this jumble of code ignites your curiosity to go digging into these libraries. There are a lot of very interesting looking function combinators (that I am still figuring out how to use) which might inspire another article.
Thanks for the clarification! ↩
TL;DR - I’m going to walk through how I rebuilt this blog using a static site generator. In the first half I walk through my decision for moving to a static site. You can skip down to the tech talk if you want.
When re-designing/building my portfolio site in 2007 I did the tried and true geek thing: I hacked up my own custom CMS using CakePHP backed by MySQL. It was pretty standard stuff, complete with a blog engine modeled after WordPress. Putting it together was a great learning experience, similar to the “15 minute blog” videos from the Rails heyday of 2005-2007. If you want to learn a language/framework, build something in it.
It was also complete overkill. As you get older, hopefully wiser and more experienced with a technology you start to realize that less is more. Over time I started to ditch the big, heavy Rails clones like CakePHP and move to smaller routers like Sinatra and SlimPHP to meet my server-side needs. When it came time to redesign my site I made a decision to separate the portfolio and blog, and also decied to build them in static HTML.
To be honest tooling was the most important. There is no way I would make a static site without a generator tool to handle menus and links. A lot of static site generators have popped up in just about every language:
I settled on Middleman, which is a self-contained Ruby Gem, because it’s written in Ruby and relies on Sinatra, both of which I already have experience with. I also liked its anything goes philosophy: you could generate a dynamic PHP site with it if you wished.
At the time that I rebuilt my blog (June 2012) Middleman 3 was still in Beta. The developers were releasing RCs rapidly, so I made a decision to develop against 3.0 instead of building the site in 2.x and then having to upgrade later. This caused some friction and headaches as documentation was out of date or I ran into Gem incompatabilities. This is to be expected when following a moving target, and the devs were fairly responsive to my cries for help. This site (the one you’re on right now) is also in a GitHub repo so you can rummage around in the source.
If you come from a Ruby web development background, Middleman will be very familiar to you. Middleman is essentially a Sinatra application that renders web pages to flat HTML files instead of dynamically serving them from a Ruby application server such as Passenger or Thin. You create templates and page content files, which Middleman renders out into static pages, pretty much like a typical dynamic site. This allows you to use just about any Ruby templating library that Sinatra already supports, which is pretty much all of them. It also lets you use partials to organize your templates. Middleman also loads in template helpers from the Padrino project to help out with links, forms, assets and more.
Since Middleman is a Sinatra app this allows you to run the application in server mode during development with live updating of content. Development is a breeze as you can tweak the site in real time and then refresh your browser on localhost
to see the changes.
YAML is the preferred method for storing data about your site (navigation, tags) and front matter for a page. If you’re coming from Jekyll/Octopress this is already familiar. You could hookup SQLite3 or MySQL if you wanted to, but then why are you building a static site?
One early quirk I ran into was with the page template system. Middleman will render pretty much anyting you enable an engine for. I found it easiest in my case to use ERB for my template files. These are the skeleton of the site itself, and you use them just like you do in a Rails site. I find HAML to abstract, and it was a lot easier for me to make my templates by hand with HTML5 Boilerplate as a starting point and then backfill the ERB markup tags in place of content. This also made it easy to load partials in the standard ERB way, so I could split out my footers and navigation into common files.
I prefer writing content in Markdown whenever possible. The way you achieve this in Middleman is to create files in your /source
directory with a stacked file extension, such as foo.html.markdown
. Middleman will render that as foo.html
using whatever Markdown engine you enabled (in my case it’s Kramdown) and drop it into your templates.
However, Markdown doesn’t support inline Ruby code, which means I couldn’t use custom helpers like entry_asset({:url => '...'})
. After poking around for a while, I finally found out that you could keep stacking file extensions onto your files for additional parsing. So my content files were now named foo.html.markdown.erb
. Middleman first parsed the ERB, then the Markdown during compilation.
Also, I had to explicitly tell the Markdown parser that my template files were ERB. So when all was said and done my Markdown configuration block looked like this:
:::ruby
set :markdown_engine, :kramdown
set :markdown, :layout_engine => :erb,
:tables => true,
:autolink => true,
:smartypants => true
One of the other attractive features of Middleman was the ability to create extensions, and a growing list of community extensions already available. Extensions are Gems which you drop into your project. Since I was making a blog, it made sense to install the Blogging extension. This extensions gave me some basic additions such as Article, Tag and Calendar pages, which are basically blog archive pages. It also provided some helpers to deal with pagination and other blog-centric issues.
Once you install the extension, you pass it a configuration block which tells the Gem where to find template files:
:::ruby
activate :blog do |blog|
blog.layout = "article"
blog.taglink = "metadata/:tag.html"
blog.tag_template = "tag.html"
blog.year_template = "calendar.html"
end
Being on a simplicity kick I decided that I didn’t need the full blown calendar/archive pages that the extension provides. A simple list of articles grouped by year would suffice. To pull this off I used one of the more powerful features of Middleman which is the ability to create what it calls dynamic pages. These are basically pages you create on the fly programmatically from within Middleman, without a content file.
In my case I setup a little block of code that gathered all of the blog articles, grouped them by year and dropped them into an array. The array is then passed into a template and the /archive.html
page is generated. This is run everytime the site is compiled, so the page is always up-to-date:
:::ruby
archive_resources = []
blog.articles.group_by {|a| a.date.year }.each do |year, year_articles|
archive_resources << {:year => year, :articles => year_articles}
end
page "/archive.html", :layout => :generic do
@archives = archive_resources
end
I use a similar process to dynamically build the Sitemap and RSS feeds for the site everytime it’s built. This automates a lot of tedious jobs that I would most likely forget to do on my own.
You can write your own Sinatra-style helpers to help out with repetetive tasks. One of my main goals for the new blog was the create a clean reading experience that looked good on a variety of platforms. This meant using a responsive design so it looked sharp on mobile devices, but it also meant I wanted it to look good when viewed on an application like Readability (full disclosure: I work for Arc90, Readability’s parent company.). Readability has a set of article publishing guidelines which explains what their parsers are looking for in your HTML.
I wanted a helper that would wrap my main article image in .entry-content-asset
classed tags so Readability and others would scoop it up as an important image. The helper is pretty simple, and uses the existing image_tag
helper to do the job:
:::ruby
# Properly format a content_entry_asset
def entry_asset(img, url = nil)
img_tag = image_tag(
img[:url],
:itemprop => "image",
:alt => img[:alt],
:title => img[:title]
)
# if we have a URL then link the image with link_to
unless url.nil?
img_tag = link_to img_tag, url
end
'<div class="entry-content-asset photo-full">' + img_tag + '</div>'
end
I could now drop the following tag at the top of my Markdown pages and the image would get wrapped in the appropriate <div>
tags:
:::ruby
entry_asset({
:url => 'http://images.darrennewton.com/backbone.jpg',
:alt => 'Spine',
:title => "Backbone.js - keep your kneebones connected to your thighbones"
})
I also needed to create a helper that would generate the Coderwall badges in the footer of this site. This one was a little more complex as I needed to hit the Coderwall API and parse out the relevant information. I thought it would be best to wrap it up as a module in the /lib
folder that was then pulled into Middleman via require "lib/coderwall_helpers"
. You can see the source for that module in the repo.
The ultimate test came when I decided to port my design portfolio over to Middleman as well. Simply copying the project over into a new repo, tweaking the templates to handle multiple large images and stripping out some things I didn’t need (such as Coderwall badges and Disqus comments) had me up and running in a very short amount of time.
Overall I had a positive experience with Middleman. There was some initial weirdness figuring out the core Sitemap resources which are at the heart of the application, but I chalk this up to being in beta and having spotty documentation. A fair amount of time was spent in the YARD docs figuring out what methods were available to build navigation.
I also spent a lot of time generating Schema.org metadata on the sites as well as alternative JSON representations of the pages to compliment traditional RSS, but that’s a topic for another blog post.
]]>In my previous blog post I linked to a demo application hosted on Heroku. Heroku is great, as you can use their free hosting to quickly get your demo up online using a variety of technologies. Unfortunately I ran into a few issues related to traffic spikes and bad weather. The cloud is great, until it’s not.
The demo application itself is a very simple Backbone.js powered photo viewer that stores references and metadata about photos in Statigram. All of the data is stored in a single PostgreSQL table. The slides for my backbone presentation also hit the application server (written in Sinatra) which in turn hit the DB. So it’s pretty chatty over the network.
So, my blog post about Backbone.js and the accompanying slides got a lot of play on Twitter and mailing lists like JavaScript Weekly. This was awesome! However, it also meant a lot of traffic to the demo application which used a dev instance of Heroku’s PostgreSQL database offering.
The dev tier of Heroku’s PostgreSQL service is free, like their free web tier. It also turns out that the dev tier can only handle 20 connections at a time, so once people start hitting the app it quickly starts timing out and throwing errors. So if a lot of folks are running the code on the slides, and also checking out the app itself then the DB is getting a lot of connections. When I first posted the demo app, it wasn’t getting that much play and seemed to run fine. More on this in a minute.
This one was beyond everyone’s control. Severe electrical storms took down the Amazon Web Service data center in Virgina, which powers much of Heroku and related services. When AWS is down, Heroku effectively goes down. Shit happens. So shortly after my blog post was up, Heroku went down taking the app with it. Not the best experience for people checking it out.
When Heroku went down due to the AWS outage I just assumed it would come back up when Heroku got everything together and brought their shared DB’s back online. Unfortunately, this coincided with the sudden popularity of the demo app. When Herkou got PostgreSQL back up, the dev DB was instantly flooded with connections and went back down. The weather trouble made it hard to tell where the problem really was, so it took a couple of days to get at the root problem (only 20 allowed connections).
Once I realized the dev tier of DB service wouldn’t cut it, I migrated to the Crane tier of service, which is a lot more robust. So far things have been cruising along fine. So if you had trouble hitting the app on Heroku, go ahead and try it now.
The downside is that a Crane instance costs $50 a month, which I consider to be a bit steep ($600 p/yr). So over the next few weeks I will work on moving things over to a MySQL DB on one of my existing Linodes. While I like Heroku quite a bit, I do find that your wallet is quickly emptied when you need to do even minimum scaling.
Heroku is a great playground, and some folks use it for their startups and production apps. If like me you use it to easily deploy proof of concept apps just be ready to handle a traffic spike if you make it into Hacker News or similar. If you need a lot of DB access you may consider setting up your own MySQL server on a cheap Linode 512.
]]>I’m currently working with the amazing folks at Arc90 on a pretty hefty project. It’s a great working environment that really stresses collaboration and learning, with weekly code and design reviews. We’re starting to use Backbone.js on a number of projects, so a quick talk was organized to explain the ins and outs to everyone. Since I had worked on some previous projects using Backbone, they asked me to do an intro.
Backbone.js is a JavaScript library (I leave it to you to figure out what it does/is), so it made sense to present an introduction in JavaScript instead of something like Keynote. After a little research the excellent Fathom.js was chosen for the task. In another blog post I’ll talk about the customizations I made to Fathom to allow me to run JavaScript in the presentation. You can checkout the slides here on Heroku.
This is a super simple introduction to Backbone.js for someone who know’s nothing about it, so it’s not comprehensive by any means. It merely introduces the concepts of Models, Collections and Views, along with Backbone.Sync’s AJAX interface.
The slides run code against a live application that is also on Heroku, which you can see here. This is a super simple client-side app that displays Instagram photos. It’s meant to illustrate simple Model/Collection handling, along with methods to assemble Views. The client-side uses Backbone.Sync to talk to a small Sinatra application on Heroku.
All of the code for the slideshow and the Instagram app are available on GitHub: DarrenN / backbone-demo
You can run the Sinatra app locally using a SQLite3 DB and it will work fine. To get it up on Heroku required switching over to their Postgres service, which you can check out on the heroku branch.
Caveat emptor: the sample app is completely unsafe as it doesn’t do any input validation or sanitizing, so you’ve been warned.
My talk was an intro to Backbone.js so I was first up. I was followed by Patrick O’Neill who showed us how to use Backbone.js with the Require.js AMD loader. Many light bulbs went off in my head during his demo. Patrick was followed by the menthol smooth Rob Madole who showed us how to use Jasmine to test our Backbone applications. I learned a lot, and hopefully passed on some knowledge to other folks. The whole thing was organized by Robert Petro, an instigator of many things at Arc90.
]]>If you’re using nginx as your webserver and need to pass URL parameters to your PHP front controller then some adjustments to your configuration file are necessary:
:::c
location /www {
try_files $uri $uri/ /www/index.php?$args;
}
I’m recenlty worked on a small PHP application built with Silex and served up with nginx. The application makes use of Twitter’s API with OAuth 1.0 (which in my opinion is very difficult to use compared to Facebook’s OAuth2 implementation.)
OAuth 1.0 requires GET/POSTing with urls like:
:::text
/www/twitter/auth?oauth_token=XXX&oauth_verifier=XXX
Everything was working fine on my laptop under Apache, but not when I moved it to my dev server under nginx. Some logging revealed that the URL parameters
were not getting picked up by Silex’s Request
class.
I took a look at my nginx.conf
and found the culprit, the redirect to the front controller was setup like so:
:::c
location /www {
try_files $uri $uri/ /www/index.php;
}
This doesn’t pass any URL parameters to index.php
, so I simply added ?$args
to the redirect:
:::c
location /www {
try_files $uri $uri/ /www/index.php?$args;
}
and booya! everything is working again.
Reference: nginx + php-fpm - where are my $_GET params?
]]>A while back on Hacker News, a.k.a Ed Weissman posted his collection of tips and advice for programmers. It was amazing stuff.
His original post didn’t have much in the way of formatting, so using a series of regexes I made a slightly more styled version with links to sections and formatted code which I posted on the Hacker News thread. Mr. Weissman didn’t seem to mind, and made note of some places where I had munged his formatting a little (which I promptly corrected.)
The reason you’re not passionate about your work is because something is missing. Identifying what is missing is your first step in determining where to go from here.
Ed Weissman
I stumbled across that file this morning while cleaning up one of my servers and decided to re-post the link for posterity’s sake here. It is loaded with gems for folks making a living or seeking to make a living as a programmer. I highly suggest reading it.
]]>I’m a command line interface kind of guy, which is funny since I do everything on Macs. Then again, I first embraced PCs with an Apple IIe, which was a CLI only experience.
One little tool I’ve had installed for a while is z by Rupa. It’s a nice little shell script that keeps tabs on the directories you’ve been cding in and out of and then lets you quickjump to them using a simple regex. So instead of remembering a long path like:
:::bash
$ cd /Users/shibuya/github/octopress
I can just do something like:
:::bash
$ z octo
If I have two directories with similar names, like website_development
and website_master
I can differentiate them like so:
:::bash
$ z web d # Will cd to /website_developement
$ z web m # Will cd to /website_master
All in all it saves me quite a few keystrokes. Check out more examples in the README.
]]>My fellow worker bees over at Gabardine.com decided to flex their design chops by lending a hand to the folks downtown at Liberty Square. While Occupy Wall Street gives us a much needed civics lesson, and the 1% continues to skim the cream off your bank accounts, Scott Grant took some of the DIY signs from the occupation and re-made them as snappy posters.
]]>A friend of mine emailed me the other day with a quick question: “What’s an easy way to convert an entire site to PDF? Are there tools for this?”
Why yes, yes there are. In fact, it’s pretty easy to do if you’re on a Mac or Linux OS using wget and wkhtmltopdf:
:::bash
$ mkdir /wget
$ wget --mirror -w 2 -p --html-extension --convert-links -P /wget http://darrenknewton.com
$ find darrenknewton.com -name '*.html' -exec wkhtmltopdf {} {}.pdf \;
$ mkdir pdfs
$ find darrenknewton.com -name '*.pdf' -exec mv {} pdfs/ \;
wget
is a great little program to grab content from the web. It’s a web Swiss Army Knife®. wkhtmltopdf
is another great piece of software which converts html
to pdf
. It can take content from the web itself, or pickup files from your desktop.
You can install both packages easily with Homebrew. If you’re a MacPorts person then you’re out of luck with wkhtmltopdf
and will need to pickup a binary.
In this example I am mirroring the site to my desktop using wget
and then doing the conversion to pdf
. I could probably rig up a way to pass urls to wkhtmltopdf
but I think making a local copy first gives me some flexibility and for a really big site I would be able to do the PDF conversion offline and at my leisure.
So first, let’s mirror the site with wget
(I’m going to use this site as an example):
:::bash
$ wget --mirror -w 2 -p --html-extension --convert-links -P /wget http://darrenknewton.com
This will spider the site and dump all of its files into /wget
, a directory I made for this demo. When you spider my site wget
will create a directory called darrenknewton.com
like this:
:::bash
$ ls -alh darrenknewton.com
drwxr-xr-x 12 shibuya staff 408B Oct 30 18:17 .
drwxr-xr-x 5 shibuya staff 170B Oct 30 18:26 ..
drwxr-xr-x 3 shibuya staff 102B Oct 30 18:17 about
-rw-r--r-- 1 shibuya staff 8.3K Oct 30 17:39 about.1.html
-rw-r--r-- 1 shibuya staff 6.6K Oct 30 09:31 atom.xml
drwxr-xr-x 6 shibuya staff 204B Oct 30 18:17 blog
-rw-r--r-- 1 shibuya staff 561B Oct 30 17:38 favicon.png
drwxr-xr-x 8 shibuya staff 272B Oct 30 17:39 images
-rw-r--r-- 1 shibuya staff 13K Oct 30 17:39 index.html
drwxr-xr-x 6 shibuya staff 204B Oct 30 17:39 javascripts
-rw-r--r-- 1 shibuya staff 22B Oct 30 09:31 robots.txt
drwxr-xr-x 3 shibuya staff 102B Oct 30 17:39 stylesheets
The .html
files are all tucked away into directories, and we don’t want to go manually searching for them, so we will let find
do it for us. The following snippet will find all of the .html
files and pass them to wkhtmltopdf
for conversion:
:::bash
$ find darrenknewton.com -name '*.html' -exec wkhtmltopdf {} {}.pdf \;
This creates PDFs alongside the html
files in the directory. So let’s grab those and dump them into their own directory.
:::bash
$ mkdir pdfs
$ find darrenknewton.com -name '*.pdf' -exec mv {} pdfs/ \;
And there you go, you should have a directory full of PDFs that look something like this.
]]>Phrap is a super-light PHP database wrapper using PDO for basic CRUD operations with MySQL. If that doesn’t sound sexy, well, that’s because it isn’t.
I often write light web-service type apps that require simple DB access, but are far to simple to spin up a big ORM like Doctrine. Phrap is a tiny library (script) that provides simple read/write/delete to the DB using PDO to handle escaping and such.
It reads, it writes, it deletes. It doesn't do associations, joins or validations, unless you feel like forking and adding. It keeps you from having to write a lot of SELECT * FROM blah blah blah
and $stmt->prepare()
every time you need something from your database, yet its stupid small so it doesn't drag on your resources (much).
This was something I needed for myself but I also used this little project to teach myself PHPUnit, and yes, I now see some of the benefits of testing your software.
Speaking of which, read the tests to get a better idea how it works.
A little Phrap:
:::php
<?php
class FileUpload extends Model
{
public $table = "files";
public $id;
public $filename;
public $userid;
public $email;
}
// Create
$dbh = new DB($database);
$file = new FileUpload($dbh);
$file->create();
$file->set(array('filename' => 'heffalump.txt', 'codswoddle' => 'phiminster'));
$file->save();
// Query
$result = $file->find('first');
$result = $file->find('all');
$result = $file->find('first', array('email' => 'dazza@email.com'));
$result = $file->query(
'SELECT * FROM files WHERE email = :email ORDER BY userid LIMIT 1',
array(':email' => 'dazza@email.com')
);
// Delete manually by id
$file->delete(1);
// Set id in object then delete it
$file->find('first');
$file->delete();
?>
]]>I spent a very frustrating hour this morning trying to get the JSON gem to work on Ruby 1.9.2 p136 installed via RVM on Snow Leopard. I thought I would post my solution to help any other unfortunate souls.
The RVM docs suggest installing a sandboxed version of Iconv like so:
:::bash
$ rvm package install iconv
This didn't work. I got archictecture errors during compile. I then tried to install the iconv
gem, which again didn't work. It couldn't seem to find the iconv
lib files.
In the end, I installed icon with Homebrew and linked it to Ruby like so:
:::bash
$ brew install libiconv
$ rvm remove 1.9.2
$ rvm install 1.9.2 --with-iconv-dir=/usr/local/Cellar/libiconv/1.13.1
This compiled Ruby with the Homebrew installed iconv
libraries and the json
gem worked just fine after.
Silly me. I stumbled across a question on Stack Overflow that was right up my alley, as it was something I had worked on before: Extract Relevant Tag/Keywords from Text block. The question posted by user593778 indicated that they wanted to use PHP or JavaScript to lift relevant keywords from a block of text.
Awesome. So I started to answer the question, outlining an approach for the poster to use to solve her problem. What I didn’t pay attention to was the fact that the user had a single point on Stack Overflow.
Stack Overflow works on a point system: it taps the lizard brain in most geeks by awarding points for answering questions, posting question, upvoting, etc. You want a high score. You’re awarded badges for hitting different thresholds in the system. Points are the carrot and the stick (you lose points on downvotes) that get people to participate and behave themselves in the community.
Having 1 point means you just logged into the system. You haven’t upvoted any other answers or answered any other questions. You haven’t even completed your user profile. This is a red flag.
Almost immediately the poster started to request more and more detail, basically asking for an implementation. We went back and forth a bit until I finally posted a full code example to the problem, at which point the poster disappeared.
So, I did their work and didn’t even get tagged as “the answer”, which means I received no points. I got played.
My solution is somewhat naive, but it works (see Gist below). It takes the input text and breaks it into an array of words, which are then filtered against a giant list of stop words.
:::php
<?php
function stopWords($text, $stopwords) {
// Remove line breaks and spaces from stopwords
$stopwords = array_map(function($x){
return trim(strtolower($x));
}, $stopwords);
// Replace all non-word chars with comma
$pattern = '/[0-9\W]/';
$text = preg_replace($pattern, ',', $text);
// Create an array from $text
$text_array = explode(",",$text);
// remove whitespace and lowercase words in $text
$text_array = array_map(function($x){
return trim(strtolower($x));
}, $text_array);
foreach ($text_array as $term) {
if (!in_array($term, $stopwords)) {
$keywords[] = $term;
}
};
return array_filter($keywords);
}
$stopwords = file('stop_words.txt');
$text = "Requirements - Working knowledge, on LAMP Environment using Linux, Apache 2, MySQL 5 and PHP 5, - Knowledge of Web 2.0 Standards - Comfortable with JSON - Hands on Experience on working with Frameworks, Zend, OOPs - Cross Browser Javascripting, JQuery etc. - Knowledge of Version Control Software such as sub-version will be preferable.";
print_r(stopWords($text, $stopwords));
?>
This is not something you would probably want to do in real time on a text of any length. You would probably schedule this as a background task.
]]>I always figure one of the best ways to learn a new language is to try out a couple of Project Euler exercises using it. I had solved this one with JavaScript previously, but using CoffeeScript let me use some more Python-style comprehensions to get to the same solution:
:::coffeescript
# Project Euler #3 -> http://projecteuler.net/index.php?section=problems&id=3
# Q: What is the largest prime factor of the number 600851475143 ?
# Generate a Sieve of Eratosthenes
# (an array loaded with prime numbers)
primes = []
sieve = (set, pos) ->
return false if pos >= set.length
if set[pos] * set[pos] < set[set.length-1]
primes = (x for x in set when x == set[pos] or x % set[pos] != 0)
return sieve(primes,pos+1)
sieve([2..200000],0)
# Factor a number into its primes
# we use our Sieve above to quickly check if number is prime
factors = []
factorial = (primes, target, count) ->
if primes[count] < target
if target % primes[count] == 0
factors.push(primes[count])
target = target / primes[count]
else
count++
if (num for num in primes when num == target).length > 0
factors.push(target)
else
factorial(primes, target, count)
factorial(primes, 600851475143, 0)
console.log(factors)
]]>I’m currently working on a large print project. It’s a membership directory for a non-profit organization and I’m laying it out with Adobe InDesign. The client handed me a CSV dump of their member database for the directory. I’ve mentioned more times than I can count how much I dislike repetition (cutting, pasting, rinse, repeat) so I cooked up some Ruby scripts to parse the CSV into various XML and plaintext formats.
To make life easier I converted the entire CSV file into an array of hashes like so:
:::ruby
require 'csv'
def csv_to_array(file_location)
csv = CSV::parse(File.open(file_location, 'r') {|f| f.read })
fields = csv.shift
csv.collect { |record| Hash[*fields.zip(record).flatten ] }
end
test = csv_to_array('test.csv')
This gets the CSV into a nice array loaded with hashes which names each entry with its corresponding field:
:::ruby
{"name"=>"Larry", "company"=>"Google"}
{"name"=>"Curly", "company"=>"Apple"}
{"name"=>"Moe", "company"=>"Microsoft"}
Getting the CSV into an array of hashes is pretty convenient. I can now rip through the hashes and do whatever formatting I need. For instance, I needed to create a plaintext file that converted each row in the CSV into a Name, Company
format followed by a line break, which could easily be pasted into an InDesign text box:
:::ruby
def csv_to_text(file_location)
out = ""
rows = csv_to_array(file_location) # get our CSV into an array of hashes
rows.each do |row|
# Use a HEREDOC to get the formatting we want
tmp = <<HERE
#{row["name"]}, #{row["company"]}
HERE
out = out + tmp
end
save_text(file_location, out) # save a text file
end
This is all pretty simple, but it saves on a lot of cutting and pasting text from fields in Excel or Numbers, the kind of thing I really dislike. You can also do a lot of text transformation during the same process. Many of the addresses in the CSV needed to be made consistent. I wanted common words like Street, Boulevard, Avenue to get transformed into common abbreviations like St., Blvd., and Ave. We need a function:
:::ruby
def process_address(data)
if data.nil?
return data
end
data.squeeze!(" ") # remove any double spaces
data.strip! # remove trailing spaces
data.gsub!(/Res: /, '')
data.gsub!(/Court/,'Ct.')
data.gsub!(/Suite/, 'Ste.')
data.gsub!(/Drive/, 'Dr.')
data.gsub!(/Road/, 'Rd.')
data.gsub!(/Street/, 'St.')
data.gsub!(/Lane/, 'Ln.')
data.gsub!(/Avenue/, 'Ave.')
data
end
Now we can pass our rows through process_address
when writing to the text file to get consistent abbreviations. (I could probably optimize that function by dropping the abbreviations into a hash and iterating through them.)
These are pretty simple little scripts, but they help to quickly automate some very tedious tasks. They're also an example of how scripting languages can help you tackle data entry issues when working with your design tools. For this same project I also needed to generate a large XML file that InDesign could parse and use to build the member directory on its own. I'll cover that chunk of code in my next post.
]]>