This post was written by Jeremy Davis, JavaScript Developer for Toptal.

CSS was designed for documents, what the “old web” was expected to contain. The emergence of pre-processors like Sass or Less shows that the community needs more than what CSS offers. With web apps getting more and more complex over time, CSS’ limitations have become increasingly visible and difficult to mitigate.

Styled-components leverages the power of a complete programming language—JavaScript—and its scoping capabilities to help structure the code into components. This helps to avoid the common pitfalls of writing and maintaining CSS for large projects. A developer can describe a component’s style with no risk of side effects.

What’s the Problem?

One advantage of using CSS is that the style is completely decoupled from the code. This means that developers and designers can work in parallel without interfering with each other.

On the other hand, styled-components makes it easier to fall in the trap of strongly coupling style and logic. Max Stoiber explains how to avoid this. While the idea of separating logic and presentation is definitely not new, one might be tempted to take shortcuts when developing React components. For example, it’s easy to create a component for a validation button that handles the click action as well as the button’s style. It takes a bit more effort to split it in two components.

The Container/Presentational Architecture

This is a pretty simple principle. Components either define how things look, or they manage data and logic. A very important aspect of presentation components is that they should never have any dependencies. They receive props and render DOM (or children) accordingly. Containers, on the other hand, know about the data architecture (state, redux, flux, etc.) but should never be responsible for display. Dan Abramov’s article is a very good and detailed explanation of this architecture.

Remembering SMACSS

Although the Scalable and Modular Architecture for CSS is a style guide for organizing CSS, the basic concept is one that is followed, for the most part automatically, by styled-components. The idea is to separate CSS into five categories:

  • Base contains all the general rules.
  • Layout’s purpose it to define the structural properties as well as various sections of content (header, footer, sidebar, content, for instance).
  • Module contains subcategories for the various logical blocks of the UI.
  • State defines modifier classes to indicate elements’ states, e.g. field in error, disabled button.
  • Theme contains color, font, and other cosmetic aspects that may be modifiable or dependent on user preference.

Keeping this separation while using styled-components is easy. Projects usually include some kind of CSS normalization or reset. This typically falls in the Base category. You may also define general font sizing, line sizing, etc. This can be done through normal CSS (or Sass/Less), or through the injectGlobal function provided by styled-components.

For Layout rules, if you use a UI framework, then it will probably define container classes, or a grid system. You can easily use those classes in conjunction with your own rules in the layout components you write.

Module is automatically followed by the architecture of styled-components, since the styles are attached to components directly, rather than described in external files. Basically, each styled component that you write will be its own module. You can write your styling code without worrying about side effects.

State will be rules you define within your components as variable rules. You simply define a function to interpolate values of your CSS attributes. If using a UI framework, you might have useful classes to add to your components as well. You will probably also have CSS pseudo-selector rules (hover, focus, etc.)

The Theme can simply be interpolated within your components. It is a good idea to define your theme as a set of variables to be used throughout your application. You can even derive colors programmatically (using a library, or manually), for instance to handle contrasts and highlights. Remember that you have the full power of a programming language at your disposal!

Bring Them Together for a Solution

It is important to keep them together, for an easier navigation experience; We don’t want to organize them by type (presentation vs. logic) but rather by functionality.

Thus, we will have a folder for all the generic components (buttons and such). The others should be organized depending on the project and its functionalities. For instance, if we have user management features, we should group all the components specific to that feature.

To apply styled-components’ container/presentation architecture to a SMACSS approach, we need an extra type of component: structural. We end up with three kinds of components; styled, structural, and container. Since styled-components decorates a tag (or component), we need this third type of component to structure the DOM. In some cases, it might be possible to allow a container component to handle the structure of sub-components, but when the DOM structure becomes complex and is required for visual purposes, it’s best to separate them. A good example is a table, where the DOM typically gets quite verbose.

Example Project

Let’s build a small app that displays recipes to illustrate these principles. We can start building a Recipes component. The parent component will be a controller. It will handle the state—in this case, the list of recipes. It will also call an API function to fetch the data.

class Recipes extends Component{
  constructor (props) {
    super(props);
    this.state = {
      recipes: []
    };
  }

  componentDidMount () {
    this.loadData()
  }

  loadData () {
    getRecipes().then(recipes => {
      this.setState({recipes})
    })
  }

  render() {
    let {recipes} = this.state

    return (
      <RecipesContainer recipes={recipes} />
    )
  }
}

It will render the list of recipes, but it does not (and should not) need to know how. So we render another component that gets the list of recipes and outputs DOM:

class RecipesContainer extends Component{
  render() {
    let {recipes} = this.props

    return (
      <TilesContainer>
        {recipes.map(recipe => (<Recipe key={recipe.id} {...recipe}/>))}
      </TilesContainer>
    )
  }
}

Here, actually, we want to make a tile grid. It may be a good idea to make the actual tile layout a generic component. So if we extract that, we get a new component that looks like this:

class TilesContainer extends Component {
  render () {
    let {children} = this.props

    return (
      <Tiles>
        {
          React.Children.map(children, (child, i) => (
            <Tile key={i}>
              {child}
            </Tile>
          ))
        }
      </Tiles>
    )
  }
}

TilesStyles.js:

export const Tiles = styled.div`
  padding: 20px 10px;
  display: flex;
  flex-direction: row;
  flex-wrap: wrap;
`

export const Tile = styled.div`
  flex: 1 1 auto;
  ...
  display: flex;

  & > div {
    flex: 1 0 auto;
  }
`

Notice that this component is purely presentational. It defines its style and wraps whatever children it receives inside another styled DOM element that defines what tiles look like. It’s a good example of what your generic presentational components will look like architecturally.

Then, we need to define what a recipe looks like. We need a container component to describe the relatively complex DOM as well as define the style when necessary. We end up with this:

class RecipeContainer extends Component {
  onChangeServings (e) {
    let {changeServings} = this.props
    changeServings(e.target.value)
  }

  render () {
    let {title, ingredients, instructions, time, servings} = this.props

    return (
      <Recipe>
        <Title>{title}</Title>
        <div>{time}</div>
        <div>Serving
          <input type="number" min="1" max="1000" value={servings} onChange={this.onChangeServings.bind(this)}/>
        </div>
        <Ingredients>
          {ingredients.map((ingredient, i) => (
            <Ingredient key={i} servings={servings}>
              <span className="name">{ingredient.name}</span>
              <span className="quantity">{ingredient.quantity * servings} {ingredient.unit}</span>
            </Ingredient>
          ))}
        </Ingredients>
        <div>
          {instructions.map((instruction, i) => (<p key={i}>{instruction}</p>))}
        </div>
      </Recipe>
    )
  }
}

Notice here that the container does some DOM generation, but it’s the only logic it contains. Remember that you can define nested styles, so you don’t need to make a styled element for each tag that requires styling. It’s what we do here for the name and quantity of the ingredient item. Of course, we could split it further and create a new component for an ingredient. That is up to you—depending on the project complexity—to determine the granularity. In this case, it is just a styled component defined along with the rest in the RecipeStyles file:

export const Recipe = styled.div`
  background-color: ${theme('colors.background-highlight')};
`;

export const Title = styled.div`
  font-weight: bold;
`

export const Ingredients = styled.ul`
  margin: 5px 0;
`

export const Ingredient = styled.li`
  & .name {
    ...
  }

  & .quantity {
    ...
  }
`

For the purpose of this exercise, I have used the ThemeProvider. It injects the theme in the props of styled components. You can simply use it as color: ${props => props.theme.core_color}, I am just using a small wrapper to protect from missing attributes in the theme:

const theme = (key) => (prop) => _.get(prop.theme, key) || console.warn('missing key', key)

You can also define your own constants in a module and use those instead. For example: color: ${styleConstants.core_color}

Pros

A perk of using styled-components is that you can use it as little as you want. You can use your favorite UI framework and add styled-components on top of it. This also means that you can easily migrate an existing project component by component. You can choose to style most of the layout with standard CSS and only use styled-components for reusable components.

Cons

Designers/style integrators will need to learn very basic JavaScript to handle variables and use them in place of Sass/Less.

They will also have to learn to navigate the project structure, although I would argue that finding the styles for a component in that component’s folder is easier than having to find the right CSS/Sass/Less file that contains the rule you need to modify.

They will also need to change their tools a bit if they want syntax highlighting, linting, etc. A good place to start is with this Atom plugin and this babel plugin.

 

This post was written by Ben Jones, Front-end Developer for Toptal.

Ben is a skilled developer who always keeps the user in mind, which allows him to see how beneficial or detrimental a development process can be. He wants to create simple but effective software to reduce workload on all sides and to make employees, employers, and customers happy.


JavaScript frameworks/libraries such as Vue can offer a fantastic user experience when browsing your site. Most offer a way of dynamically changing page content without having to send a request to the server each time.

However, there is an issue with this approach. When initially loading your website, your browser doesn’t receive a complete page to display. Instead, it gets sent a bunch of pieces to construct the page (HTML, CSS, other files) and instructions for how to put them all together (a JavaScript framework/library) It takes a measurable amount of time to put all this information together before your browser actually has something to display. It’s like being sent a bunch of books along with a flat-pack bookcase. You’d have to build the bookcase first and then fill it with the books.

The solution to this is clever: Have a version of the framework/library on the server that can build a ready-to-display page. Then send this complete page to the browser along with the ability to make further changes and still have dynamic page content (the framework/library), just like being sent a ready-made bookcase along with some books. Sure, you still have to put the books in the bookcase, but you’ve got something usable immediately.

Visual comparison of client-side and server-side rendering

Beyond the silly analogy, there are also a bunch of other advantages. For example, a page that rarely changes, such as an About Us page, doesn’t need to be recreated every single time a user asks for it. So a server can create it once and then cache it or store it somewhere for future use. These kinds of speed improvements may seem tiny, but in an environment where time until responsiveness is measured in milliseconds (or less), every little bit counts.

If you’d like more information on the advantages of SSR in a Vue environment, you should check out Vue’s own article on SSR. There is a variety of options to achieve these results, but the most popular one, which is also recommended by the Vue team, is Nuxt.

Why Nuxt.js

Nuxt.js is based off an implementation of SSR for the popular React library called Next. After seeing the advantages of this design, a similar implementation was designed for Vue called Nuxt. Those familiar with the React+Next combination will spot a bunch of similarities in design and layout of the application. However, Nuxt offers Vue-specific features to create a powerful yet flexible SSR solution for Vue.

Nuxt was updated to a production-ready 1.0 version in January 2018 and is part of an active and well-supported community. One of the great things is that building a project using Nuxt isn’t that different from building any other Vue project. In fact, it provides a bunch of features that allow you to create well-structured codebases in a reduced amount of time.

Another important thing to note is Nuxt doesn’t have to be used for SSR. It’s promoted as a framework for creating universal Vue.js applications and includes a command (nuxt generate) for creating static generated Vue applications using the same codebase. So if you’re apprehensive about diving deep into SSR, don’t panic. You can always create a static site instead while still taking advantage of Nuxt’s features.

In order to grasp the potential of Nuxt, let’s create a simple project. The final source code for this project is hosted on GitHub if you want to see it, or you can view a live version created using nuxt generate and hosted on Netlify.

Creating a Nuxt Project

To start off, let’s use a Vue project generator called vue-cli to quickly create a sample project:

# install vue-cli globally
npm install -g vue-cli

# create a project using a nuxt template
vue init nuxt-community/starter-template my-nuxt-project

After going through a couple options, this will create a project inside the folder my-nuxt-project or whatever you specified. Then we just need to install dependencies and run the server:

cd my-nuxt-project
npm install # Or yarn
npm run dev

There we go. Open your browser to localhost:3000 and your project should be running. Not much different from creating a Vue Webpack project. However, when we look at the actual structure of the app, there’s not much there, especially when compared to something like the Vue Webpack template.

Diagram of project directories and their relation to the Nuxt config file

Looking in the package.json also shows that we only have one dependency, Nuxt itself. This is because each version of Nuxt is tailored to work with specific versions of Vue, Vue-router, and Vuex and bundles them all together for you.

There is also a nuxt.config.js file at the project root. This allows you to customize a bunch of features that Nuxt provides. By default, it sets the header tags, loading bar color, and ESLint rules for you. If you’re eager to see what you can configure, here’s the documentation; we will be covering some options in this article.

So what’s so special about those directories?

Project Layout

If you browse through the directories created, all of them have an accompanying Readme stating a brief summary of what goes in that directory and often a link to the docs.

This is one benefit of using Nuxt: a default structure for your application. Any good front-end developer will structure an application similar to this, but there are many different ideas about structures, and when working on a team, some time will inevitably go into discussing or choosing this structure. Nuxt provides one for you.

Nuxt will look for certain directories and build your application for you depending on what it finds. Let’s examine these directories one by one.

Pages

This is the only required directory. Any Vue components in this directory are automatically added to vue-router based on their filenames and the directory structure. This is extremely convenient. Normally I would have a separate Pages directory anyway and have to manually register each of those components in another router file. This router file can become complex for larger projects and may need splitting to maintain readability. Instead, Nuxt will handle all of this logic for you.

To demonstrate, we can create a Vue component called about.vue inside the Pages directory. Let’s just add a simple template such as:

<template>
 <h1>About Page</h1>
</template>

When you save, Nuxt will re-generate the routes for you. Seeing as we called our component about.vue, if you navigate to /about, you should see that component. Simple.

There is one filename which is special. Naming a file index.vue will create a root route for that directory. When the project is generated, there’s already an index.vue component in the pages directory which correlates to the homepage or landing page of your site. (In the development example, this would simply be localhost:3000.)

Nuxt scans the Vue files in the pages directory and outputs the appropriate pages.

What about deeper routes? Sub-directories in the Pages directory help to structure your routes. So if we wanted a View Product page, we could structure our Pages directory something like this:

/pages
--| /products
----| index.vue
----| view.vue

Now, if we navigate to /products/view, we will see the view.vue component inside the products directory. If we navigate instead to /products, we will see the index.vuecomponent inside the products directory.

You may be asking why we didn’t just create a products.vue component in the pages directory instead like we did for the /about page. You may think the result would be the same, but there is a difference between the two structures. Let’s demonstrate this by adding another new page.

Say we wanted a seperate About page for each employee. For example, let’s create an About page for me. It should be located at /about/ben-jones. Initially, we may try structuring the Pages directory like this:

/pages
--| about.vue
--| /about
----| ben-jones.vue

When we try to access /about/ben-jones, we instead get the about.vue component, the same as /about. What’s going on here?

Interestingly, what Nuxt is doing here is generating a nested route. This structure suggests that you want a permanent /about route and anything inside that route should be nested in its own view area. In vue-router, this would be signified by specifying a <router-view />component inside the about.vue component. In Nuxt, this is the same concept except, instead of <router-view />, we simply use <nuxt />. So let’s update our about.vuecomponent to allow for nested routes:

<template>
 <div>
   <h1>About Page</h1>
   <nuxt />
 </div>
</template>

Now, when we navigate to /about, we get the about.vue component we had before, with just a title. However, when we navigate to /about/ben-jones, we instead have the title andthe ben-jones.vue component rendered where the <nuxt/> placeholder was.

This wasn’t what we initially wanted, but the idea of having an About page with a list of people that, when clicked on, fill a section on the page with their information is an interesting concept, so let’s leave it as is for now. If you did want the other option, then all we would do is restructure our directories. We’d just have to move the about.vuecomponent inside the /about directory and rename it index.vue, so the resulting structure would be:

/pages
--| /about
----| index.vue
----| ben-jones.vue

Finally, say we wanted to use route params to retrieve a specific product. For example, we want to be able to edit a product by navigating to /products/edit/64 where 64 is the product_id. We can do this the following way:

/pages
--| /products
----| /edit
------| _product_id.vue

Note the underscore at the beginning of the _product_id.vue component—this signifies a route param which is then accessible on the $route.params object or on the params object in Nuxt’s Context (more on that later). Note that the key for the param will be the component name without the initial underscore—in this case, product_id—so try to keep them unique within the project. As a result, in _product_id.vue, we may have something like:

<template>
 <h1>Editing Product {{ $route.params.product_id }}</h1>
</template>

You can start to imagine more complex layouts, which would be a pain to set up using vue-router. For example, we can combine all of the above into a route such as:

/pages
--| /categories
----| /_category_id
------| products.vue
------| /products
--------| _product_id.vue

It’s not too difficult to reason on what /categories/2/products/3 would display. We would have the products.vue component with a nested _product_id.vue component, with two route params: category_id and product_id. This is much simpler to reason on than an equivalent router config.

While we’re on the topic, one thing I tend to do in the router config is set up router guards. As Nuxt is building the router for us, this can be done instead on the component itself with beforeRouteEnter. If you want to validate route params, Nuxt provides a component method called validate. So if you wanted to check if the product_id was a number before trying to render the component, you would add the following to the script tag of _product_id.vue:

export default {
 validate ({ params }) {
   // Must be a number
   return /^\d+$/.test(params.product_id)
 }
}

Now, navigating to /categories/2/products/someproduct results in a 404 because someproduct isn’t a valid number.

That’s it for the Pages directory. Learning how to structure your routes properly in this directory is essential, so spending a little time initially is important to getting the most out of Nuxt. If you’re looking for a brief overview, it is always helpful to refer to the docs for routing.

If you’re worried about not being in control of the router, don’t be. This default setup works great for a wide variety of projects, provided they are well structured. However, there are some cases where you may need to add more routes to the router than Nuxt automatically generates for you or restructure them. Nuxt provides a way to customize the router instance in the config, allowing you to add new routes and customize generated routes. You can also edit the core functionality of the router instance, including extra options added by Nuxt. So if you do encounter an edge case, you still have the flexibility to find the appropriate solution.

Store

Nuxt can build your Vuex store based on the structure of the store directory, similar to the Pages directory. If you don’t need a store, just remove the directory. There are two modes for the store, Classic and Modules.

Classic requires you to have an index.js file in the store directory. There you need to export a function that returns a Vuex instance:

import Vuex from 'vuex'

const createStore = () => {
 return new Vuex.Store({
   state: ...,
   mutations: ...,
   actions: ...
 })
}

export default createStore

This allows you to create the store however you wish, much like using Vuex in a normal Vue project.

Modules mode also requires you to create an index.js file in the store directory. However, this file only needs to export the root state/mutations/actions for your Vuex store. The example below specifies a blank root state:

export const state = () => ({})

Then, each file in the store directory will be added to the store in its own namespace or module. For example, let’s create somewhere to store the current product. If we create a file called product.js in the store directory, then a name-spaced section of the store will be available at $store.product. Here’s a simple example of what that file may look like:

export const state = () => ({
 _id: 0,
 title: 'Unknown',
 price: 0
})

export const actions = {
 load ({ commit }) {
   setTimeout(
     commit,
     1000,
     'update',
     { _id: 1, title: 'Product', price: 99.99 }
   )
 }
}

export const mutations = {
 update (state, product) {
   Object.assign(state, product)
 }
}

The setTimeout in the load action simulates some sort of API call, which will update the store with the response; in this case, it takes one second. Now, let’s use it in the products/view page:

<template>
 <div>
   <h1>View Product {{ product._id }}</h1>
   <p>{{ product.title }}</p>
   <p>Price: {{ product.price }}</p>
 </div>
</template>

<script>
import { mapState } from 'vuex'
export default {
 created () {
   this.$store.dispatch('product/load')
 },
 computed: {
   ...mapState(['product'])
 }
}
</script>

A few things to note: Here, we are calling our fake API when the component is created. You can see that the product/load action we are dispatching is namespaced under Product. This makes it clear exactly what section of the store we are dealing with. Then, by mapping the state to a local computed property, we can easily use it in our template.

There is a problem: We see the original state for a second while the API runs. Later, we will use a solution provided by Nuxt to fix this (known as fetch).

Just to stress this again, we never had to npm install vuex, as it is already included in the Nuxt package. When you add an index.js file to the store directory, all those methods are then opened up to you automatically.

That’s the main two directories explained; the rest are much simpler.

Components

The Components directory is there to contain your reusable components such as a navigation bar, image gallery, pagination, data tables, etc. Seeing as components in the Pages directory are converted into routes, you need somewhere else to store these types of components. These components are accessible in pages or other components by importing them:

import ComponentName from ~/components/ComponentName.vue

Assets

This contains uncompiled assets and has more to do with how Webpack loads and processes files, rather than with how Nuxt works. If you’re interested, I suggest reading the guide in the Readme.

Static

This contains static files which are mapped to the root directory of your site. For example, putting an image called logo.png in this directory would make it available at /logo.png. This is good for meta files like robots.txt, favicon.ico, and other files you need available.

Layouts

Normally, in a Vue project, you have some sort of root component, normally called App.vue. Here is where you can set up your (normally static) app layout, which may include a navbar, footer, and then a content area for your vue-router. The default layout does exactly that and is provided for you in the layouts folder. Initially, all it has is a div with a <nuxt />component (which is equivalent to <router-view />) but it can be styled however you wish. For example, I’ve added a simple navbar to the example project for navigation around the various demonstration pages.

A layout can be applied to multiple pages.

You may want to have a different layout for a certain section of your app. Maybe you have some sort of CMS or admin panel that looks different. To solve this, create a new layout in the Layouts directory. As an example, let’s create an admin-layout.vue layout which just has an extra header tag and no navbar:

<template>
 <div>
   <h1>Admin Layout</h1>
   <nuxt />
 </div>
</template>

Then, we can create an admin.vue page in the Pages directory and use a property provided by Nuxt called layout to specify the name (as a string) of the layout we want to use for that component:

<template>
 <h1>Admin Page</h1>
</template>

<script>
export default {
 layout: 'admin-layout'
}
</script>

That’s all there is to it. Page components will use the default layout unless specified, but when you navigate to /admin, it now uses the admin-layout.vue layout. Of course, this layout could be shared across several admin screens if you wish. The one important thing to remember is layouts must contain a <nuxt /> element.

There’s one final thing to note about layouts. You may have noticed while experimenting that if you type an invalid URL, you are shown an error page. This error page is, in fact, another layout. Nuxt has its own error layout (source code here), but if you wanted to edit it, just create an error.vue layout and that will be used instead. The caveat here is that the error layout must not have a <nuxt /> element. You will also have access to an errorobject on the component with some basic information to display. (This is printed out in the terminal running Nuxt if you want to examine it.)

Middleware

Middleware are functions that can be executed before rendering a page or layout. There is a variety of reasons you may want to do so. Route guarding is a popular use where you could check the Vuex store for a valid login or validate some params (instead of using the validate method on the component itself). One project I worked on recently used middleware to generate dynamic breadcrumbs based on the route and params.

These functions can be asynchronous; just be careful, as nothing will be shown to the user until the middleware is resolved. They also have access to Nuxt’s Context, which I will explain later.

Plugins

This directory allows you to register Vue plugins before the application is created. This allows the plugin to be shared throughout your app on the Vue instance rather than creating it each time you import it into a component. This helps to keep the bundle sizes small.

Most major plugins have a Nuxt version that can be easily registered to the Vue instance by following their docs. However, there will be circumstances when you will be developing a plugin or need to adapt an existing plugin for this purpose. An example I’m borrowing from the docs shows how to do this for vue-notifications. First, we need to install the package:

npm install vue-notifications --save

Then create a file in the plugins directory called vue-notifications.js and include the following:

import Vue from 'vue'
import VueNotifications from 'vue-notifications'

Vue.use(VueNotifications)

Very similar to how you would register a plugin in a normal Vue environment. Then edit the nuxt.config.js file at your project root and add the following entry to the module.exports object:

plugins: ['~/plugins/vue-notifications']

That’s it. Now you can use vue-notifications throughout your app without increasing the bundle size. An example of this is at /plugin in the example project.

So that completes a rundown of the directory structure. It may seem a lot to learn, but if you’re developing a Vue app, you’re already setting up the same kind of logic. Nuxt helps to abstract away the setup and help you focus on building.

Nuxt does more than assist in development though. It supercharges your components by providing extra functionality.

Nuxt’s Supercharged Components

When I first started researching Nuxt, I kept reading about how Page components are supercharged. It sounded great, but it wasn’t immediately obvious what exactly that meant and what benefits it brings.

What it means is that all Page components have extra methods attached to them that Nuxt can use to provide additional functionality. In fact, we’ve already seen one of these earlier when we used the validate method to check params and redirect a user if they are invalid.

The two main ones used in a Nuxt project will be the asyncData and fetch methods. Both are very similar in concept, they are run asynchronously before the component is generated, and they can be used to populate the data of a component and the store. They also enable the page to be fully rendered on the server before sending it to the client even when we have to wait for some database or API call.

What’s the difference between asyncData and fetch?

  • asyncData is used to populate the data of the Page component. When you return an object, it is then merged with the output of data before rendering.
  • fetch is used to populate the Vuex Store. If you return a promise, Nuxt will wait until it is resolved before rendering.

So let’s put these to good use. Remember earlier on the /products/view page we had a problem where the initial state of the store was being displayed briefly while our fake API call was being made? One way of fixing this is having a boolean stored on the component or in the Store such as loading = true and then displaying a loading component while the API call finishes. Afterward, we would set loading = false and display the data.

Instead, let’s use fetch to populate the Store before rendering. In a new page called /products/view-async, let’s change the created method to fetch; that should work, right?

export default {
 fetch () {
   // Unfortunately the below line throws an error
   // because 'this.$store' is undefined...
   this.$store.dispatch('product/load')
 },
 computed: {...}
}

Here’s the catch: These “supercharged” methods run before the component is created, so this doesn’t point to the component and nothing on it can be accessed. So how do we access the Store here?

The Context API

Of course, there is a solution. On all of Nuxt’s methods, you are provided with an argument (normally the first) containing an extremely useful object called the Context. In this is everything you will need reference to across the app, meaning we don’t need to wait for Vue to create those references on the component first.

I would highly recommend checking out the Context docs to see what is available. Some handy ones are app, where you can access all your plugins, redirect, which can be used to change routes, error to display the error page, and some self-explanatory ones such as routequery, and store.

So, to access the Store, we can destructure the Context and extract the Store from it. We also need to make sure we return a promise so that Nuxt can wait for it to resolve before rendering the component, so we need to make a small adjustment to our Store action too.

// Component
export default {
 fetch ({ store }) {
   return store.dispatch('product/load')
 },
 computed: {...}
}

// Store Action
load ({ commit }) {
 return new Promise(resolve => {
   setTimeout(() => {
     commit('update', { _id: 1, title: 'Product', price: 99.99 })
     resolve()
   }, 1000)
 })
}

You could use async/await or other methods depending on your coding style, but the concept is the same—we’re telling Nuxt to make sure the API call finishes and the Store is updated with the result before trying the render the component. If you try navigating to /products/view-async, you will not see the flash of content where the product is in its initial state.

You can imagine how useful this can be in any Vue app even without SSR. The Context is also available to all middlewares as well as to other Nuxt methods such as NuxtServerInitwhich is a special store action that runs before the Store is initialized (an example of this is in the next section)

Considerations When Using SSR

I’m sure many (myself included) who start using a technology such as Nuxt while treating it like any other Vue project eventually hit a wall where something we know would normally work seems impossible in Nuxt. As more of these caveats are documented, it will be easier to overcome, but the main thing to consider when starting to debug is that the client and server are two separate entities.

When you access a page initially, a request is sent to Nuxt, the server builds as much as possible of that page and the rest of the app, and then the server sends it to you. Then the responsibility is on the client to continue with navigation and load chunks as it needs them.

We want the server to do as much as possible first, but sometimes it doesn’t have access to the information it needs, which results in the work being done client-side instead. Or worse, when the final content presented by the client is different from what the server expected, the client is told to rebuild it from scratch. This is a big indication that something is wrong with the application logic. Thankfully, an error will be generated in your browser’s console (in development mode) if this starts to happen.

Let’s take an example of how to solve a common issue, session management. Imagine you have a Vue app where you can log in to an account, and your session is stored using a token (JWT, for example) which you decide to keep in localStorage. When you initially access the site, you want to authenticate that token against an API, which returns some basic user info if valid and puts that information in the Store.

After reading through Nuxt’s docs, you see that there’s a handy method called NuxtServerInit which allows you to asynchronously populate the Store once on initial load. That sounds perfect! So you create your user module in the Store and add the appropriate action in the index.js file in the Store directory:

export const actions = {
 nuxtServerInit ({ dispatch }) {
   // localStorage should work, right?
   const token = localStorage.getItem('token')
   if (token) return dispatch('user/load', token)
 }
}

When you refresh the page, you get an error, localStorage is not defined. Thinking about where this is happening, it makes sense. This method is run on the server, it has no idea what is stored in localStorage on the client; in fact, it doesn’t even know what “localStorage” is! So that’s not an option.

The server tries to execute localStorage.getItem('token') but throws an error, then a caption below explaining the problem.

So what’s the solution? There are a few, actually. You can get the client to initialize the Store instead but end up losing the benefits of SSR because the client ends up doing all the work. You can set up sessions on the server and then use that to authenticate the user, but that’s another layer to set up. What’s most similar to the localStorage method is using cookies instead.

Nuxt has access to cookies because they are sent with the request from the client to the server. As with other Nuxt methods, nuxtServerInit has access to the Context, this time as the second argument because the first is reserved for the store. On the Context, we can access the req object, which stores all the headers and other information from the client request. (This will be especially familiar if you’ve used Node.js.)

So after storing the token in a cookie instead (called “token,” in this case), let’s access it on the server.

import Cookie from 'cookie'

export const actions = {
 nuxtServerInit ({ dispatch }, { req }) {
   const cookies = Cookie.parse(req.headers.cookie || '')
   const token = cookies['token'] || ''
   if (token) return dispatch('user/load', token)
 }
}

A simple solution, but one that might not be immediately obvious. Learning to think about where certain actions are happening (client, server, or both) and what they have access to takes some time but the benefits are worth it.

Deployment

Deployment with Nuxt is extremely simple. Using the same codebase, you can create an SSR app, single-page application, or static page.

Server-side Rendered App (SSR App)

This is probably what you were aiming for when using Nuxt. The basic concept for deployment here is to run the build process on whatever platform you choose and set a few configurations. I’ll use the Heroku example from the docs:

First, set up scripts for Heroku in package.json:

"scripts": {
 "dev": "nuxt",
 "build": "nuxt build",
 "start": "nuxt start",
 "heroku-postbuild": "npm run build"
}

Then set up the Heroku environment using the heroku-cli (setup instructions here:

# set Heroku variables
heroku config:set NPM_CONFIG_PRODUCTION=false
heroku config:set HOST=0.0.0.0
heroku config:set NODE_ENV=production

# deploy
git push heroku master

That’s it. Now your SSR Vue app is live ready for the world to see. Other platforms have different setups, but the process is similar. The official deployment methods currently listed are:

Single-page Application (SPA)

If you wanted to take advantage of some of the extra features Nuxt provides but avoid the server trying to render pages, then you can deploy as an SPA instead.

First, it’s best to test your application without the SSR as by default npm run dev runs with SSR on. To change that, edit the nuxt.config.js file and add the following option:

mode: 'spa',

Now, when you run npm run dev, SSR will be turned off and the application will run as an SPA for you to test. This setting also makes sure no future builds will include SSR.

If everything looks fine, then deployment is exactly the same as for an SSR app. Just remember you need to set mode: 'spa' first to let the build process know you want an SPA.

Static Pages

If you don’t want to deal with a server at all and instead want to generate pages for use with static hosting services such as Surge or Netlify, then this is the option to choose. Just bear in mind that, without a server, you won’t be able to access the req and res in the Context, so if your code relies on that, be sure to accommodate it. For example, when generating the example project, the nuxtServerInit function throws an error because it’s trying to fetch a token from the cookies in the request headers. In this project, it doesn’t matter, as that data isn’t being used anywhere, but in a real application, there would need to be an alternative way to access that data.

Once that’s sorted, deployment is easy. One thing you will probably need to change first is adding an option so that the nuxt generate command will also create a fallback file. This file will prompt the hosting service to let Nuxt handle the routing rather than the hosting service, throwing a 404 error. To do so, add the following line to nuxt.config.js:

generate: { fallback: true },

Here’s an example using Netlify, which isn’t currently in the Nuxt docs. Just bear in mind that if this is your first time using netlify-cli, you will be prompted to authenticate:

# install netlify-cli globally
npm install netlify-cli -g

# generate the application (outputs to dist/ folder)
npm run generate

# deploy
netlify deploy dist

It’s as simple as that! As mentioned at the beginning of the article, there’s a version of this project here. There’s also official deployment documentation for the following services below:

Learn More

Nuxt is updating rapidly, and this is only a small selection of the features it offers. I hope this article encourages you to try it out and see if it could help improve the capabilities of your Vue applications, allowing you to develop faster and take advantage of its powerful features.

If you’re looking for more information, then look no further than Nuxt’s official links:

Looking to up your JavaScript game? Try reading The Comprehensive Guide to JavaScript Design Patterns by fellow Toptaler Marko Mišura.

This post was written by Federico Pereiro, JavaScript Developer for Toptal.

Declarative programming is, currently, the dominant paradigm of an extensive and diverse set of domains such as databases, templating and configuration management.

In a nutshell, declarative programming consists of instructing a program on what needs to be done, instead of telling it how to do it. In practice, this approach entails providing a domain-specific language (DSL) for expressing what the user wants, and shielding them from the low-level constructs (loops, conditionals, assignments) that materialize the desired end state.

While this paradigm is a remarkable improvement over the imperative approach that it replaced, I contend that declarative programming has significant limitations, limitations that I explore in this article. Moreover, I propose a dual approach that captures the benefits of declarative programming while superseding its limitations.

CAVEATThis article emerged as the result of a multi-year personal struggle with declarative tools. Many of the claims I present here are not thoroughly proven, and some are even presented at face value. A proper critique of declarative programming would take considerable time, effort, and I would have to go back and use many of these tools; my heart is not in such an undertaking. The purpose of this article is to share a few thoughts with you, pulling no punches, and showing what worked for me. If you’ve struggled with declarative programming tools, you might find respite and alternatives. And if you enjoy the paradigm and its tools, don’t take me too seriously.

If declarative programming works well for you, I’m in no position to tell you otherwise.

You can love or hate declarative programming, but you cannot afford to ignore it.
You can love or hate declarative programming, but you cannot afford to ignore it.

The Merits Of Declarative Programming

Before we explore the limits of declarative programming, it is necessary to understand its merits.

Arguably the most successful declarative programming tool is the relational database (RDB). It might even be the first declarative tool. In any case, RDBs exhibit the two properties that I consider archetypical of declarative programming:

  • A domain specific language (DSL): the universal interface for relational databases is a DSL named Structured Query Language, most commonly known as SQL.
  • The DSL hides the lower level layer from the user: ever since Edgar F. Codd’s original paper on RDBs, it is plain that the power of this model is to dissociate the desired queries from the underlying loops, indexes and access paths that implement them.

Before RDBs, most database systems were accessed through imperative code, which is heavily dependent on low-level details such as the order of records, indexes and the physical paths to the data itself. Because these elements change over time, code often stops working because of some underlying change in the structure of the data. The resulting code is hard to write, hard to debug, hard to read and hard to maintain. I’ll go out a limb and say that most of this code was in, all likelihood, long, full of proverbial rats’ nests of conditionals, repetition and subtle, state-dependent bugs.

In the face of this, RDBs provided a tremendous productivity leap for systems developers. Now, instead of thousands of lines of imperative code, you had a clearly defined data scheme, plus hundreds (or even just tens) of queries. As a result, applications had only to deal with an abstract, meaningful and lasting representation of data, and interface it through a powerful, yet simple query language. The RDB probably raised the productivity of programmers, and companies that employed them, by an order of magnitude.

What are the commonly listed advantages of declarative programming?

Proponents of declarative programming are quick to point out the advantages. However, even they admit it comes with trade-offs.
Proponents of declarative programming are quick to point out the advantages. However, even they admit it comes with trade-offs.
  1. Readability/usability: a DSL is usually closer to a natural language (like English) than to pseudocode, hence more readable and also easier to learn by non-programmers.
  2. Succinctness: much of the boilerplate is abstracted by the DSL, leaving less lines to do the same work.
  3. Reuse: it is easier to create code that can be used for different purposes; something that’s notoriously hard when using imperative constructs.
  4. Idempotence: you can work with end states and let the program figure it out for you. For example, through an upsert operation, you can either insert a row if it is not there, or modify it if it is already there, instead of writing code to deal with both cases.
  5. Error recovery: it is easy to specify a construct that will stop at the first error instead of having to add error listeners for every possible error. (If you’ve ever written three nested callbacks in node.js, you know what I mean.)
  6. Referential transparency: although this advantage is commonly associated with functional programming, it is actually valid for any approach that minimizes manual handling of state and relies on side effects.
  7. Commutativity: the possibility of expressing an end state without having to specify the actual order in which it will be implemented.

While the above are all commonly cited advantages of declarative programming, I would like to condense them into two qualities, which will serve as guiding principles when I propose an alternative approach.

  1. A high-level layer tailored to a specific domain: declarative programming creates a high-level layer using the information of the domain to which it applies. It is clear that if we’re dealing with databases, we want a set of operations for dealing with data. Most of the seven advantages above stem from the creation of a high-level layer that is precisely tailored to a specific problem domain.
  2. Poka-yoke (fool-proofness): a domain-tailored high-level layer hides the imperative details of the implementation. This means that you commit far fewer errors because the low-level details of the system are simply not accessible. This limitation eliminates many classes of errors from your code.

Two Problems With Declarative Programming

In the following two sections, I will present the two main problems of declarative programming: separateness and lack of unfolding. Every critique needs its bogeyman, so I will use HTML templating systems as a concrete example of the shortcomings of declarative programming.

The Problem With DSLs: Separateness

Imagine that you need to write a web application with a non-trivial number of views. Hard coding these views into a set of HTML files is not an option because many components of these pages change.

The most straightforward solution, which is to generate HTML by concatenating strings, seems so horrible that you will quickly look for an alternative. The standard solution is to use a template system. Although there are different types of template systems, we will sidestep their differences for the purpose of this analysis. We can consider all of them to be similar in that the main mission of template systems is to provide an alternative to code that concatenates HTML strings using conditionals and loops, much like RDBs emerged as an alternative to code that looped through data records.

Let’s suppose we go with a standard templating system; you will encounter three sources of friction, which I will list in ascending order of importance. The first is that the template necessarily resides in a file separate from your code. Because the templating system uses a DSL, the syntax is different, so it cannot be in the same file. In simple projects, where file counts are low, the need to keep separate template files may duplicate or treble the amount of files.

I open an exception for Embedded Ruby templates (ERB), because those are integrated into Ruby source code. This is not the case for ERB-inspired tools written in other languages since those templates must also be stored as different files.

The second source of friction is that the DSL has its own syntax, one different from that of your programming language. Hence, modifying the DSL (let alone writing your own) is considerably harder. To go under the hood and change the tool, you need to learn about tokenizing and parsing, which is interesting and challenging, but hard. I happen to see this as a disadvantage.

How can one vizualise a DSL? It’s not easy, but let’s just say a DSL is a clean, shiny layer on top of low-level constructs.
How can one vizualise a DSL? It’s not easy, but let’s just say a DSL is a clean, shiny layer on top of low-level constructs.

You may ask, “Why on earth would you want to modify your tool? If you are doing a standard project, a well-written standard tool should fit the bill.” Maybe yes, maybe no.

A DSL never has the full power of a programming language. If it did, it wouldn’t be a DSL anymore, but rather a full programming language.

But isn’t that the whole point of a DSL? To not have the full power of a programming language available, so that we can achieve abstraction and eliminate most sources of bugs? Maybe, yes. However, most DSLs start simple and then gradually incorporate a growing number of the facilities of a programming language until, in fact, it becomes one. Template systems are a perfect example. Let’s see the standard features of template systems and how they correlate to programming language facilities:

  • Replace text within a template: variable substitution.
  • Repetition of a template: loops.
  • Avoid printing a template if a condition is not met: conditionals.
  • Partials: subroutines.
  • Helpers: subroutines (the only difference with partials is that helpers can access the underlying programming language and let you out of the DSL straightjacket).

This argument, that a DSL is limited because it simultaneously covets and rejects the power of a programming language, is directly proportional to the extent that the features of the DSL are directly mappable to the features of a programming language. In the case of SQL, the argument is weak because most of the things SQL offers are nothing like what you find in a normal programming language. At the other end of the spectrum, we find template systems where virtually every feature is making the DSL converge towards BASIC.

Let’s now step back and contemplate these three quintessential sources of friction, summed up by the concept of separateness. Because it is separate, a DSL needs to be located on a separate file; it is harder to modify (and even harder to write your own), and (often, but not always) needs you to add, one by one, the features you miss from a real programming language.

Separateness is an inherent problem of any DSL, no matter how well designed.

We now turn to a second problem of declarative tools, which is widespread but not inherent.

Another Problem: Lack Of Unfolding Leads To Complexity

If I had written this article a few months ago, this section would have been named Most Declarative Tools Are #@!$#@! Complex But I Don’t Know Why. In the process of writing this article I found a better way of putting it: Most Declarative Tools Are Way More Complex Than They Need To Be. I will spend the rest of this section explaining why. To analyze the complexity of a tool, I propose a measure called the complexity gap. The complexity gap is the difference between solving a given problem with a tool versus solving it in the lower level (presumably, plain imperative code) that the tool intends to replace. When the former solution is more complex than the latter, we are in presence of the complexity gap. By more complex, I mean more lines of code, code that’s harder to read, harder to modify and harder to maintain, but not necessarily all of these at the same time.

Please note that we’re not comparing the lower level solution against the best possible tool, but rather against no tool. This echoes the medical principle of “First, do no harm”.

Signs of a tool with a large complexity gap are:

  • Something that takes a few minutes to describe in rich detail in imperative terms will take hours to code using the tool, even when you know how to use the tool.
  • You feel you are constantly working around the tool rather than with the tool.
  • You are struggling to solve a straightforward problem that squarely belongs in the domain of the tool you are using, but the best Stack Overflow answer you find describes a workaround.
  • When this very straightforward problem could be solved by a certain feature (which does not exist in the tool) and you see a Github issue in the library that features a long discussion of said feature with +1s interspersed.
  • A chronic, itching, longing to ditch the tool and do the whole thing yourself inside a _ for- loop_.

I might have fallen prey to emotion here since template systems are not that complex, but this comparatively small complexity gap is not a merit of their design, but rather because the domain of applicability is quite simple (remember, we’re just generating HTML here). Whenever the same approach is used for a more complex domain (such as configuration management) the complexity gap may quickly turn your project into a quagmire.

That said, it is not necessarily unacceptable for a tool to be somewhat more complex than the lower level it intends to replace; if the tool yields code that is more readable, concise and correct, it can be worth it t. It’s an issue when the tool is several times more complex than the problem it replaces; this is flat-out unacceptable. Brian Kernighan famously stated that, “Controlling complexity is the essence of computer programming.” If a tool adds significant complexity to your project, why even use it?

The question is, why are some declarative tools so much more complex than they need be? I think it would be a mistake to blame it on poor design. Such a general explanation, a blanket ad-hominem attack on the authors of these tools, is not fair. There has to be a more accurate and enlightening explanation.

Origami time! A tool with a high-level interface to an abstract lower level has to unfold the higher level from the lower one.
Origami time! Origami time! A tool with a high-level interface to an abstract lower level has to unfold the higher level from the lower one.

My contention is that any tool that offers a high level interface to abstract a lower level must unfold this higher level from the lower one. The concept of unfolding comes from Christopher Alexander’s magnum opus, The Nature of Order – in particular Volume II. It is (hopelessly) beyond the scope of this article (not to mention my understanding) to summarize the implications of this monumental work for software design; I believe its impact will be huge in years to come. It is also beyond this article to provide a rigorous definition of unfolding processes. I will use here the concept in a heuristic way.

An unfolding process is one that, in a stepwise fashion, creates further structure without negating the existing one. At every step, each change (or differentiation, to use Alexander’s term) remains in harmony with any previous structure, when previous structure is, simply, a crystallized sequence of past changes.

Interestingly enough, Unix is a great example of the unfolding of a higher level from a lower one. In Unix, two complex features of the operative system, batch jobs and coroutines (pipes), are simply extensions of basic commands. Because of certain fundamental design decisions, such as making everything a stream of bytes, the shell being a userland programand standard I/O files, Unix is able to provide these sophisticated features with minimal complexity.

To underline why these are excellent examples of unfolding, I would like to quote a few excerpts of a 1979 paper by Dennis Ritchie, one of the authors of Unix:

On batch jobs:

… the new process control scheme instantly rendered some very valuable features trivial to implement; for example detached processes (with &) and recursive use of the shell as a command. Most systems have to supply some sort of special batch job submissionfacility and a special command interpreter for files distinct from the one used interactively.

On coroutines:

The genius of the Unix pipeline is precisely that it is constructed from the very same commands used constantly in simplex fashion.

UNIX pioneers Dennis Ritchie and Ken Thompson created a powerful demonstration of unfolding in their OS. They also saved us from a dystopian all-Windows future.
UNIX pioneers Dennis Ritchie and Ken Thompson created a powerful demonstration of unfolding in their OS. They also saved us from a dystopian all-Windows future.

This elegance and simplicity, I argue, comes from an unfolding process. Batch jobs and coroutines are unfolded from previous structures (commands run in a userland shell). I believe that because of the minimalist philosophy and limited resources of the team that created Unix, the system evolved stepwise, and as such, was able to incorporate advanced features without turning its back on to the basic ones because there weren’t enough resources to do otherwise.

In the absence of an unfolding process, the high level will be considerably more complex than necessary. In other words, the complexity of most declarative tools stem from the fact that their high level does not unfold from the low level they intend to replace.

This lack of unfoldance, if you forgive the neologism, is routinely justified by the necessity to shield the user from the lower level. This emphasis on poka-yoke (protecting the user from low level errors) comes at the expense of a large complexity gap that is self-defeating because the extra complexity will generate new classes of errors. To add insult to injury, these classes of errors have nothing to do with the problem domain but rather with the tool itself. We would not go too far if we describe these errors as iatrogenic.

Declarative templating tools, at least when applied to the task of generating HTML views, are an archetypical case of a high level that turns its back on the low level it intends to replace. How so? Because generating any non-trivial view requires logic, and templating systems, especially logic-less ones, banish logic through the main door and then smuggle some of it back through the cat door.

Note: An even weaker justification for a large complexity gap is when a tool is marketed as magic, or something that just works, the opaqueness of the low level is supposed to be an asset because a magic tool is always supposed to work without you understanding why or how. In my experience, the more magical a tool purports to be, the faster it transmutes my enthusiasm into frustration.

But what about the separation of concerns? Shouldn’t view and logic remain separate? The core mistake, here, is to put business logic and presentation logic in the same bag. Business logic certainly has no place in a template, but presentation logic exists nevertheless. Excluding logic from templates pushes presentation logic into the server where it is awkwardly accommodated. I owe the clear formulation of this point to Alexei Boronine, who makes an excellent case for it in this article.

My feeling is that roughly two thirds of the work of a template resides in its presentation logic, while the other third deals with generic issues such as concatenating strings, closing tags, escaping special characters, and so on. This is the two-faced low level nature of generating HTML views. Templating systems deal appropriately with the second half, but they don’t fare well with the first. Logic-less templates flat out turn their back on this problem, forcing you to solve it awkwardly. Other template systems suffer because they truly need to provide a non-trivial programming language so their users can actually write presentation logic.

To sum up; declarative templating tools suffer because:

  • If they were to unfold from their problem domain, they would have to provide ways to generate logical patterns;
  • A DSL that provides logic is not really a DSL, but a programming language. Note that other domains, like configuration management, also suffer from lack of “unfoldance.”

I would like to close the critique with an argument that is logically disconnected from the thread of this article, but deeply resonates with its emotional core: We have limited time to learn. Life is short, and on top of that, we need to work. In the face of our limitations, we need to spend our time learning things that will be useful and withstand time, even in the face of fast changing technology. That is why I exhort you to use tools that don’t just provide a solution but actually shed a bright light on the domain of its own applicability. RDBs teach you about data, and Unix teaches you about OS concepts, but with unsatisfactory tools that don’t unfold, I’ve always felt I was learning the intricacies of a sub-optimal solution while remaining in the dark about the nature of problem it intends to solve.

The heuristic I suggest you to consider is, value tools that illuminate their problem domain, instead of tools that obscure their problem domain behind purported features.

The Twin Approach

To overcome the two problems of declarative programming, which I have presented here, I propose a twin approach:

  • Use a data structure domain specific language (dsDSL), to overcome separateness.
  • Create a high level that unfolds from the lower level, to overcome the complexity gap.

dsDSL

A data structure DSL (dsDSL) is a DSL that is built with the data structures of a programming language. The core idea is to use the basic data structures you have available, such as strings, numbers, arrays, objects and functions, and combine them to create abstractions to deal with a specific domain.

We want to keep the power of declaring structures or actions (high level) without having to specify the patterns that implement these constructs (low level). We want to overcome the separateness between the DSL and our programming language so that we are free to use the full power of a programming language whenever we need it. This is not only possible but straightforward through dsDSLs.

If you asked me a year ago, I would have thought that the concept of dsDSL was novel, then one day, I realized that JSON itself was a perfect example of this approach! A parsed JSON object consists of data structures that declaratively represent data entries in order to get the advantages of the DSL while also making it easy to parse and handle from within a programming language. (There might be other dsDSLs out there, but so far I haven’t come across any. If you know of one, I would really appreciate your mentioning it in the comments section.)

Like JSON, a dsDSL has the following attributes:

  1. It consists of a very small set of functions: JSON has two main functions, parse and stringify.
  2. Its functions most commonly receive complex and recursive arguments: a parsed JSON is an array, or object, which usually contains further arrays and objects inside.
  3. The inputs to these functions conform to very specific forms: JSON has an explicit and strictly enforced validation schema to tell valid from invalid structures.
  4. Both the inputs and the outputs of these functions can be contained and generated by a programming language without a separate syntax.

But dsDSLs go beyond JSON in many ways. Let’s create a dsDSL for generating HTML using Javascript. Later I will touch on the issue of whether this approach may be extended to other languages (spoiler: It can definitely be done in Ruby and Python, but probably not in C).

HTML is a markup language composed of tags delimited by angle brackets (< and >). These tags may have optional attributes and contents. Attributes are simply a list of key/value attributes, and contents may be either text or other tags. Both attributes and contents are optionals for any given tag. I’m simplifying somewhat, but it is accurate.

A straightforward way to represent an HTML tag in a dsDSL is by using an array with three elements: – Tag: a string. – Attributes: an object (of the plain, key/value type) or undefined (if no attributes are necessary). – Contents: a string (text), an array (another tag) or undefined (if there’s no contents).

For example, <a href="views">Index</a> can be written as ['a', {href: 'views'}, 'Index'].

If we want to embed this anchor element into a div with class links, we can write: ['div', {class: 'links'}, ['a', {href: 'views'}, 'Index']].

To list several html tags at the same level, we can wrap them in an array:

[
   ['h1', 'Hello!'],
   ['a', {href: 'views'}, 'Index']
]

The same principle may be applied to creating multiple tags within a tag:

['body', [
   ['h1', 'Hello!'],
   ['a', {href: 'views'}, 'Index']
]]

Of course, this dsDSL won’t get us far if we don’t generate HTML from it. We need a generate function which will take our dsDSL and yield a string with HTML. So if we run generate (['a', {href: 'views'}, 'Index']), we will get the string <a href="views">Index</a>.

The idea behind any DSL is to specify a few constructs with a specific structure which is then passed to a function. In this case, the structure that makes up the dsDSL is this array, which has one to three elements; these arrays have a specific structure. If generate thoroughly validates its input (and it is both easy and important to thoroughly validate input, since these validation rules are the precise analog of a DSL’s syntax), it will tell you exactly where you went wrong with your input. After a while, you’ll start to recognize what distinguishes a valid structure in a dsDSL, and this structure will be highly suggestive of the underlying thing it generates.

Now, what are the merits of a dsDSL in contraposition to a DSL?

  • A dsDSL is an integral part of your code. It leads to lower line counts, file counts, and an overall reduction of overhead.
  • dsDSLs are easy to parse (hence easier to implement and modify). Parsing is merely iterating through the elements of an array or object. Likewise, dsDSLs are comparatively easy to design because instead of creating a new syntax (that everybody will hate) you can stick with the syntax of your programming language (which everybody hates but at least they already know it).
  • A dsDSL has all the power of a programming language. This means that a dsDSL, when properly employed, has the advantage of both a high and a low level tool.

Now, the last claim is a strong one, so I’m going to spend the rest of this section supporting it. What do I mean by properly employed? To see this in action, let’s consider an example in which we want to construct a table to display the information from an array named DATA.

var DATA = [
   {id: 1, description: 'Product 1', price: 20,  onSale: true,  categories: ['a']},
   {id: 2, description: 'Product 2', price: 60,  onSale: false, categories: ['b']},
   {id: 3, description: 'Product 3', price: 120, onSale: false, categories: ['a', 'c']},
   {id: 4, description: 'Product 4', price: 45,  onSale: true,  categories: ['a', 'b']}
]

In a real application, DATA will be generated dynamically from a database query.

Moreover, we have a FILTER variable which, when initialized, will be an array with the categories we want to display.

We want our table to:

  • Display table headers.
  • For each product, show the fields: description, price and categories.
  • Don’t print the id field, but add it as an id attribute for each row. ALTERNATE VERSION: Add an id attribute to each tr element.
  • Place a class onSale if the product is on sale.
  • Sort the products by descending price.
  • Filter certain products by category. If FILTER is an empty array, we will display all products. Otherwise, we will only display the products where the category of the product is contained within FILTER.

We can create the presentation logic that matches this requirement in ~20 lines of code:

function drawTable (DATA, FILTER) {

   var printableFields = ['description', 'price', 'categories'];

   DATA.sort (function (a, b) {return a.price - b.price});

   return ['table', [
      ['tr', dale.do (printableFields, function (field) {
         return ['th', field];
      })],
      dale.do (DATA, function (product) {
         var matches = (! FILTER || FILTER.length === 0) || dale.stop (product.categories, true, function (category) {
            return FILTER.indexOf (category) !== -1;
         });

         return matches === false ? [] : ['tr', {
            id: product.id,
            class: product.onSale ? 'onsale' : undefined
         }, dale.do (printableFields, function (field) {
            return ['td', product [field]];
         })];
      })
   ]];
}

I concede this is not a straightforward example, however, it represents a fairly simple view of the four basic functions of persistent storage, also known as CRUD. Any non-trivial web application will have views that are more complex than this.

Let’s now see what this code is doing. First, it defines a function, drawTable, to contain the presentation logic of drawing the product table. This function receives DATA and FILTER as parameters, so it can be used for different data sets and filters. drawTable fulfills the double role of partial and helper.

   var drawTable = function (DATA, FILTER) {

The inner variable, printableFields, is the only place where you need to specify which fields are printable ones, avoiding repetition and inconsistencies in the face of changing requirements.

   var printableFields = ['description', 'price', 'categories'];

We then sort DATA according to the price of its products. Notice that different and more complex sort criteria would be straightforward to implement since we have the entire programming language at our disposal.

   DATA.sort (function (a, b) {return a.price - b.price});

Here we return an object literal; an array which contains table as its first element and its contents as the second. This is the dsDSL representation of the <table> we want to create.

   return ['table', [

We now create a row with the table headers. To create its contents, we use dale.do which is a function like Array.map, but one that also works for objects. We will iterate printableFields and generate table headers for each of them:

      ['tr', dale.do (printableFields, function (field) {
         return ['th', field];
      })],

Notice that we have just implemented iteration, the workhorse of HTML generation, and we didn’t need any DSL constructs; we only needed a function to iterate a data structure and return dsDSLs. A similar native, or user-implemented function, would have done the trick as well.

Now iterate through the products contained in DATA.

      dale.do (DATA, function (product) {

We check whether this product is left out by FILTER. If FILTER is empty, we will print the product. If FILTER is not empty, we will iterate through the categories of the product until we find one that is contained within FILTER. We do this using dale.stop.

         var matches = (! FILTER || FILTER.length === 0) || dale.stop (product.categories, true, function (category) {
            return FILTER.indexOf (category) !== -1;
         });

Notice the intricacy of the conditional; it is precisely tailored to our requirement and we have total freedom for expressing it because we are in a programming language rather than a DSL.

If matches is false, we return an empty array (so we don’t print this product). Otherwise, we return a <tr> with its proper id and class and we iterate through printableFields to, well, print the fields.


         return matches === false ? [] : ['tr', {
            id: product.id,
            class: product.onSale ? 'onsale' : undefined
         }, dale.do (printableFields, function (field) {
            return ['td', product [field]];

Of course we close everything that we opened. Isn’t syntax fun?

         })];
      })
   ]];
}

Now, how do we incorporate this table into a wider context? We write a function named drawAll that will invoke all functions that generate the views. Apart from drawTable, we might also have drawHeaderdrawFooter and other comparable functions, all of which will return dsDSLs.

var drawAll = function () {
   return generate ([
      drawHeader (),
      drawTable (DATA, FILTER),
      drawFooter ()
   ]);
}

If you don’t like how the above code looks, nothing I say will convince you. This is a dsDSL at its best. You might as well stop reading the article (and drop a mean comment too because you’ve earned the right to do so if you’ve made it this far!). But seriously, if the code above doesn’t strike you as elegant, nothing else in this article will.

For those who are still with me, I would like to go back to the main claim of this section, which is that a dsDSL has the advantages of both the high and the low level:

  • The advantage of the low level resides in writing code whenever we want, getting out of the straightjacket of the DSL.
  • The advantage of the high level resides in using literals that represent what we want to declare and letting the functions of the tool convert that into the desired end state (in this case, a string with HTML).

But how is this truly different from purely imperative code? I think ultimately the elegance of the dsDSL approach boils down to the fact that code written in this way mostly consists of expressions, instead of statements. More precisely, code that uses a dsDSL is almost entirely composed of:

  • Literals that map to lower level structures.
  • Function invocations or lambdas within those literal structures that return structures of the same kind.

Code that consists mostly of expressions and which encapsulate most statements within functions is extremely succinct because all patterns of repetition can be easily abstracted. You can write arbitrary code as long as that code returns a literal that conforms to a very specific, non-arbitrary form.

A further characteristic of dsDSLs (which we don’t have time to explore here) is the possibility of using types to increase the richness and succinctness of the literal structures. I will expound on this issue on a future article.

Might it be possible to create dsDSLs beyond Javascript, the One True Language? I think that it is, indeed, possible, as long as the language supports:

  • Literals for: arrays, objects (associative arrays), function invocations, and lambdas.
  • Runtime type detection
  • Polymorphism and dynamic return types

I think this means that dsDSLs are tenable in any modern dynamic language (i.e.: Ruby, Python, Perl, PHP), but probably not in C or Java.

Walk, Then Slide: How To Unfold The High From The Low

In this section I will attempt to show a way for unfolding a high level tool from its domain. In a nutshell, the approach consists of the following steps

  1. Take two to four problems that are representative instances of a problem domain. These problems should be real. Unfolding the high level from the low one is a problem of induction, so you need real data to come up with representative solutions.
  2. Solve the problems with no tool in the most straightforward way possible.
  3. Stand back, take a good look at your solutions, and notice the common patterns among them.
  4. Find the patterns of representation (high level).
  5. Find the patterns of generation (low level).
  6. Solve the same problems with your high level layer and verify that the solutions are indeed correct.
  7. If you feel that you can easily represent all the problems with your patterns of representation, and the generation patterns for each of these instances produce correct implementations, you’re done. Otherwise, go back to the drawing board.
  8. If new problems appear, solve them with the tool and modify it accordingly.
  9. The tool should converge asymptotically to a finished state, no matter how many problems it solves. In other words, the complexity of the tool should remain constant, rather than growing with the amount of problems it solves.

Now, what the hell are patterns of representation and patterns of generation? I’m glad you asked. The patterns of representation are the patterns in which you should be able to express a problem that belongs to the domain that concerns your tool. It is an alphabet of structures that allows you to write any pattern you might wish to express within its domain of applicability. In a DSL, these would be the production rules. Let’s go back to our dsDSL for generating HTML.

The humble HTML tag is a good example of patterns of representation. Let’s take a closer look at these basic patterns.
The humble HTML tag is a good example of patterns of representation. Let’s take a closer look at these basic patterns.

The patterns of representation for HTML are the following:

  • A single tag: ['TAG']
  • A single tag with attributes: ['TAG', {attribute1: value1, attribute2: value2, ...}]
  • A single tag with contents: ['TAG', 'CONTENTS']
  • A single tag with both attributes and contents: ['TAG', {attribute1: value1, ...}, 'CONTENTS']
  • A single tag with another tag inside: ['TAG1', ['TAG2', ...]]
  • A group of tags (standalone or inside another tag): [['TAG1', ...], ['TAG2', ...]]
  • Depending on a condition, place a tag or no tag: condition ? ['TAG', ...] : [] / Depending on a condition, place an attribute or no attribute: ['TAG', {class: condition ? 'someClass': undefined}, ...]

These instances can be represented with the dsDSL notation we determined in the previous section. And this is all you need to represent any HTML you might need. More sophisticated patterns, such as conditional iteration through an object to generate a table, may be implemented with functions that return the patterns of representation above, and these patterns map directly to HTML tags.

If the patterns of representation are the structures you use to express what you want, the patterns of generation are the structures your tool will use to convert patterns of representation into the lower level structures. For HTML, these are the following:

  • Validate the input (this is actually is an universal pattern of generation).
  • Open and close tags (but not the void tags, like <input>, which are self-closing).
  • Place attributes and contents, escaping special characters (but not the contents of the <style> and <script> tags).

Believe it or not, these are the patterns you need to create an unfolding dsDSL layer that generates HTML. Similar patterns can be found for generating CSS. In fact, lith does both, in ~250 lines of code.

One last question remains to be answered: What do I mean by walk, then slide? When we deal with a problem domain, we want to use a tool that delivers us from the nasty details of that domain. In other words, we want to sweep the low level under the rug, the faster the better. The walk, then slide approach proposes exactly the opposite: spend some time on the low level. Embrace its quirks, and understand which are essential and which can be avoided in the face of a set of real, varied, and useful problems.

After walking in the low level for some time and solving useful problems, you will have a sufficiently deep understanding of their domain. The patterns of representation and generation will then arise naturally; they are wholly derived from the nature of the problem they intend to solve. You can then write code that employs them. If they work, you will be able to slide through problems where you recently had to walk through them. Sliding means many things; it implies speed, precision and lack of friction. Maybe more importantly, this quality can be felt; when solving problems with this tool, do you feel like you’re walking through the problem, or do you feel that you’re sliding through it?

Maybe the most important thing about an unfolded tool is not the fact that it frees us from having to deal with the low level. Rather, by capturing the empiric patterns of repetition in the low level, a good high level tool allows us to understand fully the domain of applicability.

An unfolded tool will not just solve a problem – it will enlighten you about the problem’s structure.

So, don’t run away from a worthy problem. First walk around it, then slide through it.

Introduction

Before digging into Node.js, you might want to read up on the benefits of using JavaScript across the stack which unifies the language and data format (JSON), allowing you to optimally reuse developer resources. As this is more a benefit of JavaScript than Node.js specifically, we won’t discuss it much here. But it’s a key advantage to incorporating Node in your stack.

JavaScript’s rising popularity has brought with it a lot of changes, and the face of web development today is dramatically different. The things that we can do on the web nowadays with JavaScript running on the server, as well as in the browser, were hard to imagine just several years ago, or were encapsulated within sandboxed environments like Flash or Java Applets.

As Wikipedia states: “Node.js is a packaged compilation of Google’s V8 JavaScript engine, the libuv platform abstraction layer, and a core library, which is itself primarily written in JavaScript.” Beyond that, it’s worth noting that Ryan Dahl, the creator of Node.js, was aiming to create real-time websites with push capability, “inspired by applications like Gmail”. In Node.js, he gave developers a tool for working in the non-blocking, event-driven I/O paradigm.

After over 20 years of stateless-web based on the stateless request-response paradigm, we finally have web applications with real-time, two-way connections.

In one sentence: Node.js shines in real-time web applications employing push technology over websockets. What is so revolutionary about that? Well, after over 20 years of stateless-web based on the stateless request-response paradigm, we finally have web applications with real-time, two-way connections, where both the client and server can initiate communication, allowing them to exchange data freely. This is in stark contrast to the typical web response paradigm, where the client always initiates communication. Additionally, it’s all based on the open web stack (HTML, CSS and JS) running over the standard port 80.

One might argue that we’ve had this for years in the form of Flash and Java Applets—but in reality, those were just sandboxed environments using the web as a transport protocol to be delivered to the client. Plus, they were run in isolation and often operated over non-standard ports, which may have required extra permissions and such.

With all of its advantages, Node.js now plays a critical role in the technology stack of many high-profile companies who depend on its unique benefits. The Node.js Foundation has consolidated all the best thinking around why enterprises should consider Node.js in a short presentation that can be found on the Node.js Foundation’s Case Studies page.

In this post, I’ll discuss not only how these advantages are accomplished, but also why you might want to use Node.js—and why not—using some of the classic web application models as examples.

How Does It Work?

The main idea of Node.js: use non-blocking, event-driven I/O to remain lightweight and efficient in the face of data-intensive real-time applications that run across distributed devices.

That’s a mouthful.

What it really means is that Node.js is not a silver-bullet new platform that will dominate the web development world. Instead, it’s a platform that fills a particular need.

What it really means is that Node.js is not a silver-bullet new platform that will dominate the web development world. Instead, it’s a platform that fills a particular need. And understanding this is absolutely essential. You definitely don’t want to use Node.js for CPU-intensive operations; in fact, using it for heavy computation will annul nearly all of its advantages. Where Node really shines is in building fast, scalable network applications, as it’s capable of handling a huge number of simultaneous connections with high throughput, which equates to high scalability.

How it works under-the-hood is pretty interesting. Compared to traditional web-serving techniques where each connection (request) spawns a new thread, taking up system RAM and eventually maxing-out at the amount of RAM available, Node.js operates on a single-thread, using non-blocking I/O calls, allowing it to support tens of thousands of concurrent connections (held in the event loop).

Diagram of traditional vs. Node.js server thread

A quick calculation: assuming that each thread potentially has an accompanying 2 MB of memory with it, running on a system with 8 GB of RAM puts us at a theoretical maximum of 4,000 concurrent connections (calculations taken from Michael Abernethy’s article “Just what is Node.js?”, published on IBM developerWorks in 2011; unfortunately, the article is not available anymore), plus the cost of context-switching between threads. That’s the scenario you typically deal with in traditional web-serving techniques. By avoiding all that, Node.js achieves scalability levels of over 1M concurrent connections, and over 600k concurrent websockets connections.

There is, of course, the question of sharing a single thread between all clients requests, and it is a potential pitfall of writing Node.js applications. Firstly, heavy computation could choke up Node’s single thread and cause problems for all clients (more on this later) as incoming requests would be blocked until said computation was completed. Secondly, developers need to be really careful not to allow an exception bubbling up to the core (topmost) Node.js event loop, which will cause the Node.js instance to terminate (effectively crashing the program).

The technique used to avoid exceptions bubbling up to the surface is passing errors back to the caller as callback parameters (instead of throwing them, like in other environments). Even if some unhandled exception manages to bubble up, tools have been developed to monitor the Node.js process and perform the necessary recovery of a crashed instance (although you probably won’t be able to recover the current state of the user session), the most common being the Forever module, or using a different approach with external system tools upstart and monit, or even just upstart.

NPM: The Node Package Manager

When discussing Node.js, one thing that definitely should not be omitted is built-in support for package management using the NPM tool that comes by default with every Node.js installation. The idea of NPM modules is quite similar to that of Ruby Gems: a set of publicly available, reusable components, available through easy installation via an online repository, with version and dependency management.

A full list of packaged modules can be found on the npm website, or accessed using the npm CLI tool that automatically gets installed with Node.js. The module ecosystem is open to all, and anyone can publish their own module that will be listed in the npm repository. A brief introduction to npm can be found in a Beginner’s Guide, and details on publishing modules in the npm Publishing Tutorial.

Some of the most useful npm modules today are:

  • express – Express.js, a Sinatra-inspired web development framework for Node.js, and the de-facto standard for the majority of Node.js applications out there today.
  • hapi – a very modular and simple to use configuration-centric framework for building web and services applications
  • connect – Connect is an extensible HTTP server framework for Node.js, providing a collection of high performance “plugins” known as middleware; serves as a base foundation for Express.
  • socket.io and sockjs – Server-side component of the two most common websockets components out there today.
  • pug (formerly Jade) – One of the popular templating engines, inspired by HAML, a default in Express.js.
  • mongodb and mongojs – MongoDB wrappers to provide the API for MongoDB object databases in Node.js.
  • redis – Redis client library.
  • lodash (underscore, lazy.js) – The JavaScript utility belt. Underscore initiated the game, but got overthrown by one of its two counterparts, mainly due to better performance and modular implementation.
  • forever – Probably the most common utility for ensuring that a given node script runs continuously. Keeps your Node.js process up in production in the face of any unexpected failures.
  • bluebird – A full featured Promises/A+ implementation with exceptionally good performance
  • moment – A lightweight JavaScript date library for parsing, validating, manipulating, and formatting dates.

The list goes on. There are tons of really useful packages out there, available to all (no offense to those that I’ve omitted here).

Examples of Where Node.js Should Be Used

CHAT

Chat is the most typical real-time, multi-user application. From IRC (back in the day), through many proprietary and open protocols running on non-standard ports, to the ability to implement everything today in Node.js with websockets running over the standard port 80.

The chat application is really the sweet-spot example for Node.js: it’s a lightweight, high traffic, data-intensive (but low processing/computation) application that runs across distributed devices. It’s also a great use-case for learning too, as it’s simple, yet it covers most of the paradigms you’ll ever use in a typical Node.js application.

Let’s try to depict how it works.

In the simplest example, we have a single chatroom on our website where people come and can exchange messages in one-to-many (actually all) fashion. For instance, say we have three people on the website all connected to our message board.

On the server-side, we have a simple Express.js application which implements two things: 1) a GET ‘/’ request handler which serves the webpage containing both a message board and a ‘Send’ button to initialize new message input, and 2) a websockets server that listens for new messages emitted by websocket clients.

On the client-side, we have an HTML page with a couple of handlers set up, one for the ‘Send’ button click event, which picks up the input message and sends it down the websocket, and another that listens for new incoming messages on the websockets client (i.e., messages sent by other users, which the server now wants the client to display).

When one of the clients posts a message, here’s what happens:

  1. Browser catches the ‘Send’ button click through a JavaScript handler, picks up the value from the input field (i.e., the message text), and emits a websocket message using the websocket client connected to our server (initialized on web page initialization).
  2. Server-side component of the websocket connection receives the message and forwards it to all other connected clients using the broadcast method.
  3. All clients receive the new message as a push message via a websockets client-side component running within the web page. They then pick up the message content and update the web page in-place by appending the new message to the board.
Diagram of client and server websockets in a Node.js application

This is the simplest example. For a more robust solution, you might use a simple cache based on the Redis store. Or in an even more advanced solution, a message queue to handle the routing of messages to clients and a more robust delivery mechanism which may cover for temporary connection losses or storing messages for registered clients while they’re offline. But regardless of the improvements that you make, Node.js will still be operating under the same basic principles: reacting to events, handling many concurrent connections, and maintaining fluidity in the user experience.

API ON TOP OF AN OBJECT DB

Although Node.js really shines with real-time applications, it’s quite a natural fit for exposing the data from object DBs (e.g. MongoDB). JSON stored data allow Node.js to function without the impedance mismatch and data conversion.

For instance, if you’re using Rails, you would convert from JSON to binary models, then expose them back as JSON over the HTTP when the data is consumed by Backbone.js, Angular.js, etc., or even plain jQuery AJAX calls. With Node.js, you can simply expose your JSON objects with a REST API for the client to consume. Additionally, you don’t need to worry about converting between JSON and whatever else when reading or writing from your database (if you’re using MongoDB). In sum, you can avoid the need for multiple conversions by using a uniform data serialization format across the client, server, and database.

QUEUED INPUTS

If you’re receiving a high amount of concurrent data, your database can become a bottleneck. As depicted above, Node.js can easily handle the concurrent connections themselves. But because database access is a blocking operation (in this case), we run into trouble. The solution is to acknowledge the client’s behavior before the data is truly written to the database.

With that approach, the system maintains its responsiveness under a heavy load, which is particularly useful when the client doesn’t need firm confirmation of a the successful data write. Typical examples include: the logging or writing of user-tracking data, processed in batches and not used until a later time; as well as operations that don’t need to be reflected instantly (like updating a ‘Likes’ count on Facebook) where eventual consistency (so often used in NoSQL world) is acceptable.

Data gets queued through some kind of caching or message queuing infrastructure (e.g., RabbitMQZeroMQ) and digested by a separate database batch-write process, or computation intensive processing backend services, written in a better performing platform for such tasks. Similar behavior can be implemented with other languages/frameworks, but not on the same hardware, with the same high, maintained throughput.

Diagram of a database batch-write in Node.js with message queuing

In short: with Node, you can push the database writes off to the side and deal with them later, proceeding as if they succeeded.

DATA STREAMING

In more traditional web platforms, HTTP requests and responses are treated like isolated event; in fact, they’re actually streams. This observation can be utilized in Node.js to build some cool features. For example, it’s possible to process files while they’re still being uploaded, as the data comes in through a stream and we can process it in an online fashion. This could be done for real-time audio or video encoding, and proxying between different data sources (see next section).

PROXY

Node.js is easily employed as a server-side proxy where it can handle a large amount of simultaneous connections in a non-blocking manner. It’s especially useful for proxying different services with different response times, or collecting data from multiple source points.

An example: consider a server-side application communicating with third-party resources, pulling in data from different sources, or storing assets like images and videos to third-party cloud services.

Although dedicated proxy servers do exist, using Node instead might be helpful if your proxying infrastructure is non-existent or if you need a solution for local development. By this, I mean that you could build a client-side app with a Node.js development server for assets and proxying/stubbing API requests, while in production you’d handle such interactions with a dedicated proxy service (nginx, HAProxy, etc.).

BROKERAGE – STOCK TRADER’S DASHBOARD

Let’s get back to the application level. Another example where desktop software dominates, but could be easily replaced with a real-time web solution is brokers’ trading software, used to track stocks prices, perform calculations/technical analysis, and create graphs/charts.

Switching to a real-time web-based solution would allow brokers to easily switch workstations or working places. Soon, we might start seeing them on the beach in Florida.. or Ibiza.. or Bali.

APPLICATION MONITORING DASHBOARD

Another common use-case in which Node-with-web-sockets fits perfectly: tracking website visitors and visualizing their interactions in real-time.

You could be gathering real-time stats from your user, or even moving it to the next level by introducing targeted interactions with your visitors by opening a communication channel when they reach a specific point in your funnel. (If you’re interested, this idea is already being productized by CANDDi.)

Imagine how you could improve your business if you knew what your visitors were doing in real-time—if you could visualize their interactions. With the real-time, two-way sockets of Node.js, now you can.

SYSTEM MONITORING DASHBOARD

Now, let’s visit the infrastructure side of things. Imagine, for example, an SaaS provider that wants to offer its users a service-monitoring page (e.g., GitHub’s status page). With the Node.js event-loop, we can create a powerful web-based dashboard that checks the services’ statuses in an asynchronous manner and pushes data to clients using websockets.

Both internal (intra-company) and public services’ statuses can be reported live and in real-time using this technology. Push that idea a little further and try to imagine a Network Operations Center (NOC) monitoring applications in a telecommunications operator, cloud/network/hosting provider, or some financial institution, all run on the open web stack backed by Node.js and websockets instead of Java and/or Java Applets.

Note: Don’t try to build hard real-time systems in Node (i.e., systems requiring consistent response times). Erlang is probably a better choice for that class of application.

Where Node.js Can Be Used

SERVER-SIDE WEB APPLICATIONS

Node.js with Express.js can also be used to create classic web applications on the server-side. However, while possible, this request-response paradigm in which Node.js would be carrying around rendered HTML is not the most typical use-case. There are arguments to be made for and against this approach. Here are some facts to consider:

Pros:

  • If your application doesn’t have any CPU intensive computation, you can build it in Javascript top-to-bottom, even down to the database level if you use JSON storage Object DB like MongoDB. This eases development (including hiring) significantly.
  • Crawlers receive a fully-rendered HTML response, which is far more SEO-friendly than, say, a Single Page Application or a websockets app run on top of Node.js.

Cons:

  • Any CPU intensive computation will block Node.js responsiveness, so a threaded platform is a better approach. Alternatively, you could try scaling out the computation [*].
  • Using Node.js with a relational database is still quite a pain (see below for more detail). Do yourself a favour and pick up any other environment like Rails, Django, or ASP.Net MVC if you’re trying to perform relational operations.
[*] An alternative to these CPU intensive computations is to create a highly scalable MQ-backed environment with back-end processing to keep Node as a front-facing ‘clerk’ to handle client requests asynchronously.

Where Node.js Shouldn’t Be Used

SERVER-SIDE WEB APPLICATION W/ A RELATIONAL DB BEHIND

Comparing Node.js with Express.js against Ruby on Rails, for example, there is a clean decision in favour of the latter when it comes to relational data access.

Relational DB tools for Node.js are still in their early stages; they’re rather immature and not as pleasant to work with. On the other hand, Rails automagically provides data access setup right out of the box together with DB schema migrations support tools and other Gems (pun intended). Rails and its peer frameworks have mature and proven Active Record or Data Mapper data access layer implementations, which you’ll sorely miss if you try to replicate them in pure JavaScript.[*]

Still, if you’re really inclined to remain JS all-the-way (and ready to pull out some of your hair), keep an eye on Sequelize and Node ORM2—both are still immature, but they may eventually catch up.

[*] It’s possible and not uncommon to use Node solely as a front-end, while keeping your Rails back-end and its easy-access to a relational DB.

HEAVY SERVER-SIDE COMPUTATION/PROCESSING

When it comes to heavy computation, Node.js is not the best platform around. No, you definitely don’t want to build a Fibonacci computation server in Node.js. In general, any CPU intensive operation annuls all the throughput benefits Node offers with its event-driven, non-blocking I/O model because any incoming requests will be blocked while the thread is occupied with your number-crunching.

As stated previously, Node.js is single-threaded and uses only a single CPU core. When it comes to adding concurrency on a multi-core server, there is some work being done by the Node core team in the form of a cluster module [ref: http://nodejs.org/api/cluster.html]. You can also run several Node.js server instances pretty easily behind a reverse proxy via nginx.

With clustering, you should still offload all heavy computation to background processes written in a more appropriate environment for that, and having them communicate via a message queue server like RabbitMQ.

Even though your background processing might be run on the same server initially, such an approach has the potential for very high scalability. Those background processing services could be easily distributed out to separate worker servers without the need to configure the loads of front-facing web servers.

Of course, you’d use the same approach on other platforms too, but with Node.js you get that high reqs/sec throughput we’ve talked about, as each request is a small task handled very quickly and efficiently.

Conclusion

We’ve discussed Node.js from theory to practice, beginning with its goals and ambitions, and ending with its sweet spots and pitfalls. When people run into problems with Node, it almost always boils down to the fact that blocking operations are the root of all evil—99% of Node misuses come as a direct consequence.

In Node, blocking operations are the root of all evil—99% of Node misuses come as a direct consequence.

Remember: Node.js was never created to solve the compute scaling problem. It was created to solve the I/O scaling problem, which it does really well.

Why use Node.js? If your use case does not contain CPU intensive operations nor access any blocking resources, you can exploit the benefits of Node.js and enjoy fast and scalable network applications. Welcome to the real-time web.

This post was written by Tomislav Capan, JavaScript Developer for Toptal.

What is AngularJS?

AngularJS is a JavaScript MVC framework developed by Google that lets you build well structured, easily testable, and maintainable front-end applications.

And Why Should I Use It?

If you haven’t tried AngularJS yet, you’re missing out. The framework consists of a tightly integrated toolset that will help you build well structured, rich client-side applications in a modular fashion—with less code and more flexibility.

AngularJS extends HTML by providing directives that add functionality to your markup and allow you to create powerful dynamic templates. You can also create your own directives, crafting reusable components that fill your needs and abstracting away all the DOM manipulation logic.

It also implements two-way data binding, connecting your HTML (views) to your JavaScript objects (models) seamlessly. In simple terms, this means that any update on your model will be immediately reflected in your view without the need for any DOM manipulation or event handling (e.g., with jQuery).

Angular provides services on top of XHR that dramatically simplify your code and allow you to abstract API calls into reusable services. With that, you can move your model and business logic to the front-end and build back-end agnostic web apps.

Finally, I love Angular because of its flexibility regarding server communication. Like most JavaScript MVC frameworks, it lets you work with any server-side technology as long as it can serve your app through a RESTful web API. But Angular also provides services on top of XHR that dramatically simplify your code and allow you to abstract API calls into reusable services. As a result, you can move your model and business logic to the front-end and build back-end agnostic web apps. In this post, we’ll do just that, one step at a time.

So, Where Do I Begin?

First, let’s decide the nature of the app we want to build. In this guide, we’d prefer not to spend too much time on the back-end, so we’ll write something based on data that’s easily attainable on the Internet—like a sports feed app!

Since I happen to be a huge fan of motor racing and Formula 1, I’ll use an autosport API service to act as our back-end. Luckily, the guys at Ergast are kind enough to provide a free motorsport API that will be perfect for us.

For a sneak peak at what we’re going to build, take a look at the live demo. To prettify the demo and show off some Angular templating, I applied a Bootstrap theme from WrapBootstrap, but seeing as this article isn’t about CSS, I’ll just abstract it away from the examples and leave it out.

Getting Started Tutorial

Let’s kickstart our example app with some boilerplate. I recommend the angular-seed project as it not only provides you with a great skeleton for bootstrapping, but also sets the ground for unit testing with Karma and Jasmine (we won’t be doing any testing in this demo, so we’ll just leave that stuff aside for now; see Part 2 of this tutorial for more info on setting up your project for unit and end-to-end testing).

EDIT (May 2014): Since I wrote this tutorial, the angular-seed project has gone through some heavy changes (including the additon of Bower as package manager). If you have any doubts about how to deploy the project, take a quick look at the first section of their reference guide. In Part 2 of ths tutorial, Bower, among other tools, is covered in greater detail.

OK, now that we’ve cloned the repository and installed the dependencies, our app’s skeleton will look like this:

angularjs tutorial - start with the skeleton

Now we can start coding. As we’re trying to build a sports feed for a racing championship, let’s begin with the most relevant view: the championship table.

the championship table

Given that we already have a drivers list defined within our scope (hang with me – we’ll get there), and ignoring any CSS (for readability), our HTML might look like:

<body ng-app="F1FeederApp" ng-controller="driversController">
  <table>
    <thead>
      <tr><th colspan="4">Drivers Championship Standings</th></tr>
    </thead>
    <tbody>
      <tr ng-repeat="driver in driversList">
        <td>{{$index + 1}}</td>
        <td>
          <img src="img/flags/{{driver.Driver.nationality}}.png" />
          {{driver.Driver.givenName}}&nbsp;{{driver.Driver.familyName}}
        </td>
        <td>{{driver.Constructors[0].name}}</td>
        <td>{{driver.points}}</td>
      </tr>
    </tbody>
  </table>
</body>

The first thing you’ll notice in this template is the use of expressions (“{{“ and “}}”) to return variable values. In AngularJS, expressions allow you to execute some computation in order to return a desired value. Some valid expressions would be:

  • {{ 1 + 1 }}
  • {{ 946757880 | date }}
  • {{ user.name }}

Effectively, expressions are JavaScript-like snippets. But despite being very powerful, you shouldn’t use expressions to implement any higher-level logic. For that, we use directives.

Understanding Basic Directives

The second thing you’ll notice is the presence of ng-attributes, which you wouldn’t see in typical markup. Those are directives.

At a high level, directives are markers (such as attributes, tags, and class names) that tell AngularJS to attach a given behaviour to a DOM element (or transform it, replace it, etc.). Let’s take a look at the ones we’ve seen already:

  • The ng-app directive is responsible for bootstrapping your app defining its scope. In AngularJS, you can have multiple apps within the same page, so this directive defines where each distinct app starts and ends.
  • The ng-controller directive defines which controller will be in charge of your view. In this case, we denote the driversController, which will provide our list of drivers (driversList).
  • The ng-repeat directive is one of the most commonly used and serves to define your template scope when looping through collections. In the example above, it replicates a line in the table for each driver in driversList.

Adding Controllers

Of course, there’s no use for our view without a controller. Let’s add driversController to our controllers.js:

angular.module('F1FeederApp.controllers', []).
controller('driversController', function($scope) {
    $scope.driversList = [
      {
          Driver: {
              givenName: 'Sebastian',
              familyName: 'Vettel'
          },
          points: 322,
          nationality: "German",
          Constructors: [
              {name: "Red Bull"}
          ]
      },
      {
          Driver: {
          givenName: 'Fernando',
              familyName: 'Alonso'
          },
          points: 207,
          nationality: "Spanish",
          Constructors: [
              {name: "Ferrari"}
          ]
      }
    ];
});

You may have noticed the $scope variable we’re passing as a parameter to the controller. The $scopevariable is supposed to link your controller and views. In particular, it holds all the data that will be used within your template. Anything you add to it (like the driversList in the above example) will be directly accessible in your views. For now, let’s just work with a dummy (static) data array, which we will replace later with our API service.

Now, add this to app.js:

angular.module('F1FeederApp', [
  'F1FeederApp.controllers'
]);

With this line of code, we actually initialize our app and register the modules on which it depends. We’ll come back to that file (app.js) later on.

Now, let’s put everything together in index.html:

<!DOCTYPE html>
<html>
<head>
  <title>F-1 Feeder</title>
</head>

<body ng-app="F1FeederApp" ng-controller="driversController">
  <table>
    <thead>
      <tr><th colspan="4">Drivers Championship Standings</th></tr>
    </thead>
    <tbody>
      <tr ng-repeat="driver in driversList">
        <td>{{$index + 1}}</td>
        <td>
          <img src="img/flags/{{driver.Driver.nationality}}.png" />
          {{driver.Driver.givenName}}&nbsp;{{driver.Driver.familyName}}
        </td>
        <td>{{driver.Constructors[0].name}}</td>
        <td>{{driver.points}}</td>
      </tr>
    </tbody>
  </table>
  <script src="bower_components/angular/angular.js"></script>
  <script src="bower_components/angular-route/angular-route.js"></script>
  <script src="js/app.js"></script>
  <script src="js/services.js"></script>
  <script src="js/controllers.js"></script>
</body>
</html>

Modulo minor mistakes, you can now boot up your app and check your (static) list of drivers.

Note: If you need help debugging your app and visualizing your models and scope within the browser, I recommend taking a look at the awesome Batarang plugin for Chrome.

Loading Data From the Server

Since we already know how to display our controller’s data in our view, it’s time to actually fetch live data from a RESTful server.

To facilitate communication with HTTP servers, AngularJS provides the $http and $resource services. The former is but a layer on top of XMLHttpRequest or JSONP, while the latter provides a higher level of abstraction. We’ll use $http.

To abstract our server API calls from the controller, let’s create our own custom service which will fetch our data and act as a wrapper around $http by adding this to our services.js:

angular.module('F1FeederApp.services', []).
  factory('ergastAPIservice', function($http) {

    var ergastAPI = {};

    ergastAPI.getDrivers = function() {
      return $http({
        method: 'JSONP', 
        url: 'http://ergast.com/api/f1/2013/driverStandings.json?callback=JSON_CALLBACK'
      });
    }

    return ergastAPI;
  });

With the first two lines, we create a new module (F1FeederApp.services) and register a service within that module (ergastAPIservice). Notice that we pass $http as a parameter to that service. This tells Angular’s dependency injection engine that our new service requires (or depends on) the $http service.

In a similar fashion, we need to tell Angular to include our new module into our app. Let’s register it with app.js, replacing our existing code with:

angular.module('F1FeederApp', [
  'F1FeederApp.controllers',
  'F1FeederApp.services'
]);

Now, all we need to do is tweak our controller.js a bit, include ergastAPIservice as a dependency, and we’ll be good to go:

angular.module('F1FeederApp.controllers', []).
  controller('driversController', function($scope, ergastAPIservice) {
    $scope.nameFilter = null;
    $scope.driversList = [];

    ergastAPIservice.getDrivers().success(function (response) {
        //Dig into the responde to get the relevant data
        $scope.driversList = response.MRData.StandingsTable.StandingsLists[0].DriverStandings;
    });
  });

Now reload the app and check out the result. Notice that we didn’t make any changes to our template, but we added a nameFilter variable to our scope. Let’s put that variable to use.

Filters

Great! We have a functional controller. But it only shows a list of drivers. Let’s add some functionality by implementing a simple text search input which will filter our list. Let’s add the following line to our index.html, right below the <body> tag:

<input type="text" ng-model="nameFilter" placeholder="Search..."/>

We are now making use of the ng-model directive. This directive binds our text field to the $scope.nameFiltervariable and makes sure that its value is always up-to-date with the input value. Now, let’s visit index.html one more time and make a small adjustment to the line that contains the ng-repeat directive:

<tr ng-repeat="driver in driversList | filter: nameFilter">

This line tells ng-repeat that, before outputting the data, the driversList array must be filtered by the value stored in nameFilter.

At this point, two-way data binding kicks in: every time a value is input in the search field, Angular immediately ensures that the $scope.nameFilter that we associated with it is updated with the new value. Since the binding works both ways, the moment the nameFilter value is updated, the second directive associated to it (i.e., the ng-repeat) also gets the new value and the view is updated immediately.

Reload the app and check out the search bar.

app search bar

Notice that this filter will look for the keyword on all attributes of the model, including the ones we´re not using. Let’s say we only want to filter by Driver.givenName and Driver.familyName: First, we add to driversController, right below the $scope.driversList = []; line:

$scope.searchFilter = function (driver) {
    var keyword = new RegExp($scope.nameFilter, 'i');
    return !$scope.nameFilter || keyword.test(driver.Driver.givenName) || keyword.test(driver.Driver.familyName);
};

Now, back to index.html, we update the line that contains the ng-repeat directive:

<tr ng-repeat="driver in driversList | filter: searchFilter">

Reload the app one more time and now we have a search by name.

Routes

Our next goal is to create a driver details page which will let us click on each driver and see his/her career details.

First, let’s include the $routeProvider service (in app.js) which will help us deal with these varied application routes. Then, we’ll add two such routes: one for the championship table and another for the driver details. Here’s our new app.js:

angular.module('F1FeederApp', [
  'F1FeederApp.services',
  'F1FeederApp.controllers',
  'ngRoute'
]).
config(['$routeProvider', function($routeProvider) {
  $routeProvider.
	when("/drivers", {templateUrl: "partials/drivers.html", controller: "driversController"}).
	when("/drivers/:id", {templateUrl: "partials/driver.html", controller: "driverController"}).
	otherwise({redirectTo: '/drivers'});
}]);

With that change, navigating to http://domain/#/drivers will load the driversController and look for the partial view to render in partials/drivers.html. But wait! We don’t have any partial views yet, right? We’ll need to create those too.

Partial Views

AngularJS will allow you to bind your routes to specific controllers and views.

But first, we need to tell Angular where to render these partial views. For that, we’ll use the ng-view directive, modifying our index.html to mirror the following:

<!DOCTYPE html>
<html>
<head>
  <title>F-1 Feeder</title>
</head>

<body ng-app="F1FeederApp">
  <ng-view></ng-view>
  <script src="bower_components/angular/angular.js"></script>
  <script src="bower_components/angular-route/angular-route.js"></script>
  <script src="js/app.js"></script>
  <script src="js/services.js"></script>
  <script src="js/controllers.js"></script>
</body>
</html>

Now, whenever we navigate through our app routes, Angular will load the associated view and render it in place of the <ng-view> tag. All we need to do is create a file named partials/drivers.html and put our championship table HTML there. We’ll also use this chance to link the driver name to our driver details route:

<input type="text" ng-model="nameFilter" placeholder="Search..."/>
<table>
<thead>
  <tr><th colspan="4">Drivers Championship Standings</th></tr>
</thead>
<tbody>
  <tr ng-repeat="driver in driversList | filter: searchFilter">
    <td>{{$index + 1}}</td>
    <td>
      <img src="img/flags/{{driver.Driver.nationality}}.png" />
      <a href="#/drivers/{{driver.Driver.driverId}}">
	  	{{driver.Driver.givenName}}&nbsp;{{driver.Driver.familyName}}
	  </a>
	</td>
    <td>{{driver.Constructors[0].name}}</td>
    <td>{{driver.points}}</td>
  </tr>
</tbody>
</table>

Finally, let’s decide what we want to show in the details page. How about a summary of all the relevant facts about the driver (e.g., birth, nationality) along with a table containing his/her recent results? To do that, we add to services.js:

angular.module('F1FeederApp.services', [])
  .factory('ergastAPIservice', function($http) {

    var ergastAPI = {};

    ergastAPI.getDrivers = function() {
      return $http({
        method: 'JSONP', 
        url: 'http://ergast.com/api/f1/2013/driverStandings.json?callback=JSON_CALLBACK'
      });
    }

    ergastAPI.getDriverDetails = function(id) {
      return $http({
        method: 'JSONP', 
        url: 'http://ergast.com/api/f1/2013/drivers/'+ id +'/driverStandings.json?callback=JSON_CALLBACK'
      });
    }

    ergastAPI.getDriverRaces = function(id) {
      return $http({
        method: 'JSONP', 
        url: 'http://ergast.com/api/f1/2013/drivers/'+ id +'/results.json?callback=JSON_CALLBACK'
      });
    }

    return ergastAPI;
  });

This time, we provide the driver’s ID to the service so that we retrieve the information relevant solely to a specific driver. Now, we modify controllers.js:

angular.module('F1FeederApp.controllers', []).

  /* Drivers controller */
  controller('driversController', function($scope, ergastAPIservice) {
    $scope.nameFilter = null;
    $scope.driversList = [];
    $scope.searchFilter = function (driver) {
        var re = new RegExp($scope.nameFilter, 'i');
        return !$scope.nameFilter || re.test(driver.Driver.givenName) || re.test(driver.Driver.familyName);
    };

    ergastAPIservice.getDrivers().success(function (response) {
        //Digging into the response to get the relevant data
        $scope.driversList = response.MRData.StandingsTable.StandingsLists[0].DriverStandings;
    });
  }).

  /* Driver controller */
  controller('driverController', function($scope, $routeParams, ergastAPIservice) {
    $scope.id = $routeParams.id;
    $scope.races = [];
    $scope.driver = null;

    ergastAPIservice.getDriverDetails($scope.id).success(function (response) {
        $scope.driver = response.MRData.StandingsTable.StandingsLists[0].DriverStandings[0]; 
    });

    ergastAPIservice.getDriverRaces($scope.id).success(function (response) {
        $scope.races = response.MRData.RaceTable.Races; 
    }); 
  });

The important thing to notice here is that we just injected the $routeParams service into the driver controller. This service will allow us to access our URL parameters (for the :id, in this case) using $routeParams.id.

Now that we have our data in the scope, we only need the remaining partial view. Let’s create a file named partials/driver.html and add:

<section id="main">
  <a href="./#/drivers"><- Back to drivers list</a>
  <nav id="secondary" class="main-nav">
    <div class="driver-picture">
      <div class="avatar">
        <img ng-show="driver" src="img/drivers/{{driver.Driver.driverId}}.png" />
        <img ng-show="driver" src="img/flags/{{driver.Driver.nationality}}.png" /><br/>
        {{driver.Driver.givenName}} {{driver.Driver.familyName}}
      </div>
    </div>
    <div class="driver-status">
      Country: {{driver.Driver.nationality}}   <br/>
      Team: {{driver.Constructors[0].name}}<br/>
      Birth: {{driver.Driver.dateOfBirth}}<br/>
      <a href="{{driver.Driver.url}}" target="_blank">Biography</a>
    </div>
  </nav>

  <div class="main-content">
    <table class="result-table">
      <thead>
        <tr><th colspan="5">Formula 1 2013 Results</th></tr>
      </thead>
      <tbody>
        <tr>
          <td>Round</td> <td>Grand Prix</td> <td>Team</td> <td>Grid</td> <td>Race</td>
        </tr>
        <tr ng-repeat="race in races">
          <td>{{race.round}}</td>
          <td><img  src="img/flags/{{race.Circuit.Location.country}}.png" />{{race.raceName}}</td>
          <td>{{race.Results[0].Constructor.name}}</td>
          <td>{{race.Results[0].grid}}</td>
          <td>{{race.Results[0].position}}</td>
        </tr>
      </tbody>
    </table>
  </div>

</section>

Notice that we’re now putting the ng-show directive to good use. This directive will only show the HTML element if the expression provided is true (i.e., neither false, nor null). In this case, the avatar will only show up once the driver object has been loaded into the scope by the controller.

Finishing Touches

Add in a bunch of CSS and render your page. You should end up with something like this:

page rendered with CSS

You’re now ready to fire up your app and make sure both routes are working as desired. You could also add a static menu to index.html to improve the user’s navigation capabilities. The possibilities are endless.

EDIT (May 2014): I’ve received many requests for a downloadable version of the code that we build in this tutorial. I’ve therefore decided to release it here (stripped of any CSS). However, I really do not recommend downloading it, since this guide contains every single step you need to build the same application with your own hands, which will be a much more useful and effective learning exercise.

This article is written by Raoni Boaventura and originally published at Toptal

About the author:  Raoni is an experienced software developer and who has contributed to a wealth of projects using Ruby on Rails, JavaScript, and PHP on top of many other programming languages and frameworks. He is an excellent problem solver, and a great communicator as both a team member and a team lead.