A hidden state through time

A one-number recurrent network reads a sentence and must remember whether the subject was one key or many.

Plain recurrent network reading "The key to the cabinet in the hall by the door". At the blank the state is 0.19, so it predicts "is". The gradient reaching the subject is 0.1.
Word	Hidden state h	Gradient from the blank
The	0.00	0.038
key+1	0.76	0.1
to	0.60	0.17
the	0.49	0.25
cabinet	0.41	0.34
in	0.36	0.43
the	0.31	0.53
hall	0.27	0.63
by	0.24	0.75
the	0.21	0.87
door	0.19	1
___	State above 0, so it predicts "is". The right verb is "is".

Weights set by hand: the subject adds +1 for one key or −1 for several, and every other word adds 0. A trained network learns its own weights and would also have to learn to ignore the other nouns.

Challenges

0 of 4 complete

With the plain RNN and the long sentence, make the network still predict the right verb with a state above 0.5 at the blank. (not yet)
Still with the plain RNN on the long sentence, find a weight that sends at least 4% of the gradient back to the subject. (not yet)
Remove the squashing and make the gradient at the subject more than 1,000 times the gradient at the blank. (not yet)
Switch to the gated cell and, on the long sentence, keep the state at the blank above 0.5 while at least half the gradient reaches the subject. (not yet)

Try this

Press Read it on the medium sentence and watch the state fade after “key”.
Raise the recurrent weight above 1 and compare the state at the blank with the gradient reaching the subject.
Switch to No squashing with a weight above 1, then to Gated cell with the forget gate near 1.

The demo's network in a few linesPython with NumPy

import numpy as np

# The demo's long sentence: "The key that my grandmother hid ... last winter ___".
# Input is +1 at the singular subject ("key") and 0 for every other word.
x = np.zeros(20)
x[1] = 1.0

def run(w, u=1.0, cell="tanh"):
    h, states, factors = 0.0, [], []
    for xt in x:
        if cell == "tanh":
            h = np.tanh(w * h + u * xt)
            factors.append(w * (1 - h**2))  # d h_t / d h_(t-1)
        else:  # no squashing
            h = w * h + u * xt
            factors.append(w)
        states.append(h)
    return np.array(states), np.array(factors)

def gradient_to(step, factors):
    # Backpropagation through time: multiply the factors of every later step.
    return np.prod(factors[step + 1:])

for cell, w in [("tanh", 0.5), ("tanh", 1.0), ("tanh", 1.5), ("linear", 1.5)]:
    states, factors = run(w, cell=cell)
    print(f"{cell:6} w={w}: state at blank {states[-1]:8.4f}, "
          f"gradient reaching 'key' {gradient_to(1, factors):.2e}")

# Gated cell: the input gate opens only at the subject, the forget gate is f.
f = 0.97
c = 0.0
for xt in x:
    c = f * c + (1.0 if xt != 0 else 0.0) * xt
print(f"gated  f={f}: state at blank {np.tanh(c):8.4f}, "
      f"gradient reaching 'key' {f ** 18:.2e}")

tanh   w=0.5: state at blank   0.0000, gradient reaching 'key' 3.17e-06
tanh   w=1.0: state at blank   0.2679, gradient reaching 'key' 4.12e-02
tanh   w=1.5: state at blank   0.8586, gradient reaching 'key' 8.18e-08
linear w=1.5: state at blank 1477.8919, gradient reaching 'key' 1.48e+03
gated  f=0.97: state at blank   0.5212, gradient reaching 'key' 5.78e-01