Tuesday, 9 October 2018

Merged gulp tasks never fire `end` event

I've got a gulp task that loops through a folder looking for sub folders and outputs a JavaScript file based upon the contents of each folder. Below is a more visual example.

  • src
    • assets
      • scripts
      • critical
        • loadCSS.init.js
      • legacy
        • flexibility.init.js
        • picturefill.init.js
      • modern
        • connectivity.js
        • layzr.init.js
        • menu_list.js
        • scroll-hint.init.js
        • slideout.init.js
        • swiper.init.js
      • service-worker
        • service-worker.js

becomes:

  • dev
    • assets
      • scripts
      • critical.js
      • legacy.js
      • modern.js
      • service-worker.js

This is achieved by reading the contents of the src/assets/scripts directory, then running a loop against each folder (critical, legacy, modern, service-worker) and sending the contents of each folder to a Gulp tasks which get merged together with merge-stream.

All this works great, except that once the tasks are merged back together, I want to trigger a notification if the compilation succeeded. If I try to pipe anything to the merged streams, it doesn't work. It just returns the merged streams, and never continues on.

If I un-promisify my PROCESS_SCRIPTS function and don't use merge-stream (i.e. only processing one manually specified folder), it works fine, so I'm at a loss as to what's going on.

Here's my full task:

module.exports = {
    scripts(gulp, plugins, ran_tasks, on_error) {
        // task-specific plugins
        const ESLINT  = require("gulp-eslint");
        const WEBPACK = require("webpack-stream");

        // process scripts
        const PROCESS_SCRIPTS = (js_directory, destination_file_name = "modern.js", compare_file_name = "modern.js", source = [global.settings.paths.src + "/assets/scripts/*.js"]) => {
            return new Promise((resolve, reject) => {
                const WEBPACK_CONFIG = {
                    mode: "development",
                };

                // update webpack config for the current target destination and file name
                WEBPACK_CONFIG.mode   = plugins.argv.dist ? "production" : WEBPACK_CONFIG.mode;
                WEBPACK_CONFIG.output = {
                    filename: destination_file_name
                };

                const TASK = gulp.src(source)
                    // prevent breaking on error
                    .pipe(plugins.plumber({errorHandler: on_error}))
                    // check if source is newer than destination
                    .pipe(plugins.newer(js_directory + "/" + compare_file_name))
                    // lint all scripts
                    .pipe(ESLINT())
                    // print lint errors
                    .pipe(ESLINT.format())
                    // run webpack
                    .pipe(WEBPACK(WEBPACK_CONFIG))
                    // generate a hash and add it to the file name
                    .pipe(plugins.hash({template: "<%= name %>.<%= hash %><%= ext %>"}))
                    // output scripts to compiled directory
                    .pipe(gulp.dest(js_directory))
                    // generate a hash manfiest
                    .pipe(plugins.hash.manifest(".hashmanifest-scripts", {
                        deleteOld: true,
                        sourceDir: js_directory
                    }))
                    // output hash manifest in root
                    .pipe(gulp.dest("."))
                    // reject after errors
                    .on("error", () => {
                        return reject(TASK);
                    })
                    // return the task after completion
                    .on("end", () => {
                        return resolve(TASK);
                    });
            });
        };

        // scripts task, lints, concatenates, & compresses JS
        return new Promise ((resolve) => {
            // set JS directory
            const JS_DIRECTORY = plugins.argv.dist ? global.settings.paths.dist + "/assets/scripts" : global.settings.paths.dev + "/assets/scripts";

            // set the source directory
            const SOURCE_DIRECTORY = global.settings.paths.src + "/assets/scripts";

            // set up an empty merged stream
            const MERGED_STREAMS = plugins.merge();
            // get the script source folder list
            const SCRIPT_FOLDERS = plugins.fs.readdirSync(SOURCE_DIRECTORY);
            // get the script destination file list
            const SCRIPT_FILES   = plugins.fs.existsSync(JS_DIRECTORY) ? plugins.fs.readdirSync(JS_DIRECTORY) : false;

            // process all the script folders
            const PROCESS_SCRIPT_FOLDERS = () => {
                return new Promise((resolve) => {
                    const FOLDER_NAME = SCRIPT_FOLDERS.shift();
                    const FILE_NAME   = SCRIPT_FILES ? SCRIPT_FILES.find((name) => {
                        return name.match(new RegExp(FOLDER_NAME + ".[a-z0-9]{8}.js"));
                    }) : FOLDER_NAME + ".js";

                    // process all scripts, update the stream
                    return PROCESS_SCRIPTS(JS_DIRECTORY, FOLDER_NAME + ".js", FILE_NAME, SOURCE_DIRECTORY + "/" + FOLDER_NAME + "/**/*").then((processed) => {
                        MERGED_STREAMS.add(processed);
                        return resolve();
                    });
                }).then(() => SCRIPT_FOLDERS.length > 0 ? PROCESS_SCRIPT_FOLDERS() : resolve()); // loop again if folders remain, otherwise resolve
            };

            return PROCESS_SCRIPT_FOLDERS().then(() => {
                // wrap up
                return MERGED_STREAMS // STOPS HERE!
                    // prevent breaking on error
                    .pipe(plugins.plumber({
                        errorHandler: on_error,
                    }))
                    // notify that task is complete, if not part of default or watch
                    .pipe(plugins.gulpif(gulp.seq.indexOf("scripts") > gulp.seq.indexOf("default"), plugins.notify({
                        title:   "Success!",
                        message: "Scripts task complete!",
                        onLast:  true,
                    })))
                    // push task to ran_tasks array
                    .on("data", () => {
                        if (ran_tasks.indexOf("scripts") < 0) {
                            ran_tasks.push("scripts");
                        }
                    })
                    .on("end", () => {
                        console.log("end!"); // NEVER FIRES!
                        return resolve();
                    });
            });

        });
    }
};

Also visible on my GitHub: https://github.com/JacobDB/new-site/blob/master/gulp-tasks/scripts.js


EDIT: I've tried a few things, I'll detail them here.

  1. console.log("hello world") never fires after MERGED_STREAMS.on("data"), MERGED_STREAMS.on("error"), or MERGED_STREAMS.on("end").
  2. Moving const MERGED_STREAMS = plugins.merge(); to a module-level variable (i.e. just after const WEBPACK = require("webpack-stream")) does not change the outcome.
  3. Doing #2 and then using MERGED_STREAMS.add(gulp.src(source) ...) instead of adding the stream after the promise completes does not change the outcome, except when leaving .pipe(gulp.dist(".")), which is required to output a .hashmanifest, and always marks the task as ran.
  4. Disabling webpack, hash, or eslint, in any combination does not make a difference.
  5. Changing PROCESS_SCRIPTS from returning a promise to return the stream, then processing each folder as individual variables, then merging them manually does appear to correctly trigger the task as ran, but webpack can only be run once, so it only outputs one file – critical.hash.js. Note: I haven't tested this method in conjunction with disabling hash, which may be causing it to be marked as correctly ran if .hashmanifest is always being output.
  6. Splitting the linting step and the webpack step in to separate task kind of causes the task to be correctly marked as ran, but only if the lint task is not a promise, which results in unexpected end of stream errors in the console.


from Merged gulp tasks never fire `end` event

No comments:

Post a Comment