Not today...

comments

Snippet

Getting rid of gulp bunch of dependencies

Tagged dev , javascript

Recently Nodejs environment broke due the removal from npm of a small library (11 SLOC): leftpad. As it hit the world and broke a bunch of projects and CIs, I asked myself if my projects contains so much dependencies that if one break, everything collapse.

The problem

For developing my frontend I use a tool which I really like: Gulp. The issue there, is that for working with multiple building process involved a lot of glue and third party libraries. Here is an example: https://github.com/gulpjs/gulp/blob/master/docs/recipes/browserify-uglify-sourcemap.md Just for using browserify (which is another great tool). Each of those libraries involved other dependencies and so on until we download the whole internet to perform the most simple tasks.

The solution ?

According to this, I started looking at those libraries in order to see if I can implement them with a reduced number of dependencies and lines of code. The answer is: yes, and moreover it is quite simple and helps me learned some new things.

Here is the goal of the exercise:

  • rewrite vinyl-source-stream, merge-stream and vinyl-buffer
  • fix browserify in the gulp environment, according to this issue

The implementation

To perform the implementation I will only keep the through2 module for better creating streams (don’t worry as it is a requirements of all the gulp plugins it will not add a new dependency) and gulp util which will allow me to have some helpers to deal with gulp (as through2 it will not create new dependencies as it is a requirements for basically all the gulp plugins).

I will create a simple utils.js file aside my gulpfile.js to store those implementations.

Here is the requirementents of the file:

var through = require('through2');
var gutil = require('gulp-util');

vinyl-source-stream

According to its presentation this module only provide a convenient wrapper in order to link legacy nodejs streams and gulp implementation of streams vinyl.

Here is the code I use:

module.exports.source = function(filename) {

  // this is basically a stream, I will use the javascript scoping for
  // convenience
  var ins = through();

  // here is the piping to the previous stream, we need an object stream as we
  // will push a vinyl file in it.
  // The use will be the following:
  // someLegacyStream.pipe(utils.source()).pipe(whateverInGulpWorld);
  return through.obj(function(chunk, _, cb) {

    // Checks if we have initialized the stream with a vinyl file
    // basically, this just happen once at startup of streaming.
    if (!this._ins) {
      this._ins = ins; // this is just a convenient way of keeping a state

      // Create a vinyl file and pass "ins" stream as a content,
      // and an optional filename
      this.push(new gutil.File({contents: ins, path: filename}));
    }

    // push the chunk into our new stream in order to unify output
    ins.push(chunk);

    // as this is asynchronous just notify the system that we handled
    // the chunk
    cb();

  }, function() {
    // close the stream by pushing "null"
    ins.push(null);
    this.push(null);
  });
};

Okay, so, we replace a whole module by ~14 SLOC, additionally I get back some understanding on how nodejs streams work, and on gulp plugins implementation.

This is a good start! Let’s continue this way

merge-streams

This module simply merge multiple streams in a single one, this allow the developper to compose with multiple inputs.


// the function treats arguments as a list of streams
module.exports.merge = function(/* streams... */) {

  var sources = []; // keep a track of streams merged
  var output  = gutil.noop(); // this will be the output,
                              // a simple stream which does nothing

  // this function will be called when unpiping and when a source stream ends
  function remove(source) {
    // remove the stream from sources array
    sources = sources.filter(function(it) { return it !== source; });

    // if it is the last stream opened and if the output is not yet closed,
    // we close it
    if (!sources.length && output.readable) output.end();
  }

  // when a stream is unpiped we remove it
  output.on('unpipe', remove);

  // for each stream (arguments is not a regular array this is why we use
  // this syntax)
  Array.prototype.slice.call(arguments).forEach(function(source) {

    // add the stream to our array of sources
    sources.push(source);

    // bind the remove function to the end event of the source
    source.once('end', remove.bind(null, source));

    // pipe the stream to our output, and let it open (the output)
    // even when the stream ends (in order to handle the others)
    source.pipe(output, {end: false});
  });

  return output;
};

Here again we replace an entire module with few lines of code (~15 SLOC).

vinyl-buffer

This final module is also part of the vinyl utilitaries. It takes the chunks from a stream and return them as a nodejs buffer. Like the others, this one is quite simple and only requires to know a bit of node internal operations and libraries.


module.exports.buffer = function() {

  // like the others we will create a stream object
  return through.obj(function(file, _, cb) {

    var that = this; // keep an internal reference of this across js scoping
    var bufs = []; // array of buffer we will populate

    // if it is already a buffer or it contains nothing, just push and finish
    if (file.isNull() || file.isBuffer()) {
      that.push(file);
      return cb();
    }

    // otherwise we take the content of the stream and we pipe it
    file.contents.pipe(through.obj(
      function(data, _, cb) {
        // create a new buffer with data if it is not and push it to our array
        bufs.push(Buffer.isBuffer(data) ? data : new Buffer(data));
        cb();
      },
      function() {
        // when we have retrieved all the chunks, create a copy of file
        file = file.clone();
        // and replace the content with only one huge buffer
        file.contents = Buffer.concat(bufs);

        // push it and that's it
        that.push(file);
        cb();
      }
    ));
  });
};

We are still under 20 lines (we start to have a pattern here… just trolling). This part is done !

browserify error handling with gulp

As I mentionned it quickly at the beginning I also want to fix an issue I have with browserify and gulp.

I finally found the solution on stack overflow and I am surprised that not a lot of people ran in this problem before.

I also use the gulp.src syntax to retrieve files instead of loading the globbing module which does exactly the same things (one dependency removed, Yay \o/).

Here is how I implemented this:

// we need the browserify module
var browserify = require('browserify');

// we create a utils.browserify kind of plugin here, which can take options
module.exports.browserify = function(options) {

  // initialize browserify with options
  var b = browserify(options || {debug: true});

  // here we create a stream which will be pluggable through a pipe
  // in order to avoid spending resources we will use it like this
  // gulp.src('**/*.js', {read: false}).pipe(utils.browserify()).pipe(whatever)
  // note the "read: false" which will avoid reading file content and only
  // provide vinyl file object (not opened) to the stream
  var s = through.obj(

    // here we only retrieve the files provided by the stream
    function(file, _, cb) {
      b.add(file.path);
      cb();
    },

    // and at flush, we bundle the result through browserify
    function() {
      b.bundle()

      // on error we provide a helpful message to the user through gutil.log
      .on('error', function(err) {
        var message = err.annotated || err.toString();
        gutil.log(new gutil.PluginError('browserify', message).toString());

        // and here is the magic not provided before, we notice gulp that
        // the stream as ended, and it can continue to watch our files states
        s.emit('end');
      })

      // when data is provided by the stream we simply push it into the
      // returned one
      .on('data', function(chunk) { s.push(chunk); })

      // do not forget to pass the end event also
      .on('end', function() { s.emit('end'); });
    }
  );

  return s;
};

This one was quite more complicated but it finally works and I am happy to stay in the “gulp world” and do not provide any tricky thing.

Conclusion

This experience was really interesting, reducing code dependencies, gaining power on underlying nodejs concepts, fixing some bugs also. This was not a waste of time and I really enjoyed it.

I also will not make any comment on node ecosystem because I think everything has already been told. But this exercise proved that some libraries are really not complicated and can be reimplemented in order to avoid bad surprise in the future.