JavaScript: Iterators and Generators

Aug 25, 2019 by Nicklas Envall

The goal of this article is to give you a foundation so you can start working with generators. To do so, we will start looking at iterators, why we start looking at iterators will become clear later.

Iterables and Iterators in JavaScript

An iterator is an object that let us iterate through a sequence of values. Objects that have an iterator are called iterables. An iterable has a function called Symbol.iterator (ES6 symbol value) that returns an iterator.

One important thing with iterators is that they always must have a next() method. The next() method returns an object containing the result of one iteration (one step in the sequence) which looks like {done: boolean, value: anyvalue}. If the property done is true then we've reached the end of the sequence. If the property done is false then we have more values to iterate through.

In JavaScript, an array has its iterator, allow me to show you:

let arr = [1, 2, 3];             
let it = arr[Symbol.iterator](); // get iterator

it.next(); // { value: 1, done: false }
it.next(); // { value: 2, done: false }
it.next(); // { value: 3, done: false }
it.next(); // { value: undefined, done: true }

Here we get a new value after each iteration and information whether or not we should continue. You might wonder why the property done is not true on the third result, it's the last element after all. The answer is that the array iterator is built that way, soon we will build some iterators where how it works it should become more clear.

But why do we even use these iterators, can't we just use a for-of loop? No, we can't, since the for-of loop uses the iterator and its next method to iterate through the values. So the for-of requires the object to be iterable. Let's see what happens when we remove the arrays iterator:

let arr = [1, 2, 3];

for (var x of arr) {
    console.log(x);
}
// 1, 2, 3

arr[Symbol.iterator] = undefined;

for (var x of arr) {
    console.log(x); 
}
// TypeError: arr is not iterable

If you've ever tried to use a for-of with an object you have created, this just might be the reason for the confusion that arises when the following does not work:

let stuff = {0: 1, 1: 2};

for (var x of stuff) {
    console.log(x); 
}
// TypeError: stuff is not iterable

How to make objects iterable in JavaScript

So how do we make our objects iterable? Well now that we know the following:

An iterable must have a property called Symbol.iterator that returns an iterator.
The iterator must have a next() method.
The next() method returns a result object of a step in the sequence.
The result returned from next() looks like {done: boolean, value: anyvalue}.

By knowing this, we may implement our iterator:

let iterable = {
    [Symbol.iterator]: function () {
        return {
            counter: 0,
            next: function () {
                if (this.counter < 5) {
                    this.counter++;
                    return {done: false, value: this.counter};
                }
                return {done: true, value: undefined};
            }
        }
    },
}

let it = iterable[Symbol.iterator]();
it.next(); // { done: false, value: 1 }
it.next(); // { done: false, value: 2 }
it.next(); // { done: false, value: 3 }
it.next(); // { done: false, value: 4 }
it.next(); // { done: false, value: 5 }
it.next(); // { done: true, value: undefined }


for (var x of iterable) {
    console.log(x);
}
// 1, 2, 3, 4, 5

for (var x of iterable) {
    console.log(x);
}
// 1, 2, 3, 4, 5

As you see above, we maintain a state (counter) and use the next method to take a step further in the sequence while also returning a value.

Why two for-of loops? I displayed two for-of loops on purpose, which will become clear in the next example. Firstly we must be aware that we can create iterables with different approaches. In the example above we get a new object each time we call iterable[Symbol.iterator](), but we could also put the next method directly in the iterable object, but that would mean the state would be maintained:

let iterable = {
    counter: 0,
    [Symbol.iterator]: function () { return this },
    next: function () {
        if (this.counter < 5) {
            this.counter++;
            return {done: false, value: this.counter};
        }
        return {done: true, value: undefined};
    }
}

for (var x of iterable) {
    console.log(x);
}
// 1, 2, 3, 4, 5

for (var x of iterable) {
    console.log(x);
}
// undefined

You might like the look of example 2 better than 1, but as you see you can't loop it twice, so be aware of that. By using closure, we can still achieve a similar syntax while still also being able to use the for-of multiple times:


function iterable () {
    var counter = 0;
    
    return {
        [Symbol.iterator] : function () { return this },
        next: function () {
            
            if (counter < 5) {
                counter++;
                return {done: false, value: counter};
            }
                
            return {done: true, value: undefined};
        }
    }
}

for (var x of iterable()) {
    console.log(x);
}
// 1, 2, 3, 4, 5

for (var x of iterable()) {
    console.log(x);
}
// 1, 2, 3, 4, 5

Generators in JavaScript

We can pause and start generators at will. During its execution, we can pause it, which allows us to execute some other code outside the generator. Later we can start the generator again to continue its execution. Being able to pause and re-start a generator allows it to be able to return multiple values, unlike normal functions. A normal function returns a value once we invoke it (default is undefined). With a generator, we must request a new value each time we want one.

To define a generator function we add an asterisk (*) after the function keyword.

function* FruitGenerator() {
    yield "apple";
    yield "orange";
}

const it = FruitGenerator();
it.next(); // { value: "apple", done: false }
it.next(); // { value: "orange", done: false }
it.next(); // { value: undefined, done: true }

for (var x of FruitGenerator()) {
    console.log(x); // apple, orange
}

When we call FruitGenerator() we will get an iterator back. This iterator will be used to control the execution of the generator. Each time we call it.next() the generator's code gets executed until it reaches the keyword yield, then it returns a result of the iteration. The value property of the iteration result gets set to the value in front of the yield keyword. After that, the generator waits until we call it.next() and when we do it re-continues from where it was and executes until it reaches yet another yield or a return statement (no explicit written return statement implicitly returns undefined).

You can use a return statement if you want to return an iteration result object that has its done property set to true. In the following example, we see how only "apple" is printed:

function* FruitGenerator() {
    yield "apple";
    return "orange";
}

const it = FruitGenerator();
it.next(); // {value: "apple", done: false}
it.next(); // {value: "orange", done: true}

for (var x of FruitGenerator()) {
    console.log(x); // apple
}

What we up until now have covered is the foundation for generators. Now in the next sections, we'll look at more "advanced examples" or things we can do with generators if you will.

Generators working together with `yield*`

Putting an asterisk (*) after the yield keyword allows us to yield to another generator, delegating to another generator.

function* MelonGenerator() {
    yield "Honeydew";
    yield "Cantaloupe";
}

function* OrangeGenerator() {
    yield "Clementine";
    yield "Mandarine";
}

// note: yield* not yield
function* FruitGenerator() {
    yield* OrangeGenerator();
    yield* MelonGenerator();
}

for (var fruit of FruitGenerator()) {
    console.log(fruit);
}
// Clementine
// Mandarine
// Honeydew
// Cantaloupe

Communicate with the generator

We can use the next() method to pass arguments to a generator while also resuming its execution.

function* add() {
    var x = 1 + (yield);
    return x;
}

const it = add();
it.next();  // { value: undefined, done: false }
it.next(2); // { value: 3, done: true }

The first call to next will start the generator, then it'll get paused when it meets the first yield. The first call also returns the yielded value, which in this case is undefined. With the first call, we are basically "setting up" the opportunity to send a value in the subsequent next() which we do with next(2).

If you do not want to return undefined on yield, you may do the following:

function* add() {
    var x = 1 + (yield "im a string");
    return x;
}

const it = add();
it.next();  // { value: "im a string", done: false }
it.next(2); // { value: 3, done: true }

Generators and Promises

If you want, you can refresh your memory on how promises and asynchronous programming in JavaScript work, before continuing. Anyways, let's look at what the point of using promises and generators together is all about.

In its essence we combine generators and promises to make our asynchronous code look more synchronous:

// fetch("url") returns a promise

run(function* logFetch() {
    try {
        const res = yield fetch("url");
        console.log(res);
    }
    catch(error) {
        // do something
    }
})

function run(generator) {
    const iterator = generator();
    
    const iteration = iterator.next();
    const promise = iteration.value;

    promise
    .then(res => iterator.next(res))
    .catch(err => iterator.throw(err));
}

Let's breakdown what is happening here. The purpose of the run function is to communicate with the generator. In run we iterate one step and then logFetch returns a promise and gets suspended. Now in our run function, we have a promise which we attach a callback to that will ensure to give logFetch the result of the fetch and start it again which will console.log the result.

The following would've happened without this approach:

const res = fetch("url");
console.log(res); // Promise { <state>: "pending" }

Now with the help of a generator, we can synchronously write:

// ...
        const res = yield fetch("url");
        console.log(res); // Response { ... }
// ...

The example's run function is hardcoded to suit the logFetch generator function, which is not good. If you actually would use generators and promises together, you would create a run that works for other generator functions aswell.