word
sp
SO
NOW
WE'RE
BEING
RECORDED
sp
ALL
VERY
EXCITING
I
HOPE
THEY
CAN
HEAR
US
sp
SO
IF
WE
CAN
HAVE
sp
SOME
GOOD
EXAMPLES
OF
THE
ENGLISH
LANGUAGE
sp
PLEASE
sp
{LG}
sp
OKAY
sp
UP
TO
NOW
sp
WE'VE
sp
BEEN
COVERING
sp
ONE
PART
OF
sp
THE
NEURAL
NETWORKS
ERM
IN
FACT
sp
WE
sp
ONLY
COV
sp
COVERED
sp
ONE
PART
sp
OF
sp
ONE
NETWORK
IN
FACT
sp
{OOV}
ER
sp
THAT
A
sp
FORM
OF
sp
CONTINUOUS
NETWORK
ER
sp
BECAUSE
IT
sp
HAS
sp
CONTINUOUS
sp
WEIGHTS
sp
VARI
sp
LOT
OF
VARIATIONS
IN
DIFFERENT
SORTS
OF
NEURAL
NETWORKS
WE'VE
HAD
DIFFERENT
SORTS
OF
WEIGHTS
DIFFERENCE
SORTS
OF
INPUTS
sp
AND
sp
LEARNING
RULES
sp
AS
YOU'LL
SEE
sp
WHAT
I
WANT
TO
COVER
NOW
IS
sp
A
sp
TYPE
OF
NETWORK
THAT
WE
STUDY
A
LOT
sp
IN
THE
COMPUTER
ARCHITECTURE
sp
GROUP
sp
AND
I'VE
BEEN
WORKING
WITH
sp
FOR
YEARS
AND
YEARS
AND
YEARS
sp
ERM
CALLED
sp
THE
N
TUPLE
NETWORK
sp
N
sp
TUPLE
METHOD
sp
IT
sp
IN
NEURAL
NETWORK
TERMS
sp
IS
A
sp
BINARY
WEIGHTED
IT
sp
MEANS
THAT
sp
THE
WEIGHTS
ARE
TYPICALLY
sp
BINARY
sp
IN
sp
THE
NETWORK
SO
YOU
CAN
ALSO
USE
LOTS
sp
OF
DIFFERENT
LEARNING
RULES
sp
COMPAR
sp
COMPARED
TO
THE
sp
ER
NETWORKS
sp
YOU
SEE
sp
QUITE
sp
DIFFERENT
sp
HOWEVER
sp
EVEN
THOUGH
IT
IS
A
NEURAL
NETWORK
sp
HOWEVER
ERM
sp
YOU
CAN
SEE
IT
IS
A
PACK
RECOGNITION
TECHNIQUE
AND
IN
FACT
IT
WAS
sp
FIRST
DEVELOPED
sp
IN
NINETEEN
sp
FIFTY
NINE
sp
AS
sp
THE
N
TUPLE
sp
METHOD
ER
sp
BY
sp
SOME
sp
CHAP
sp
{GAP_ANONYMIZATION}
BROWNING
sp
ERM
I
sp
THINK
{GAP_ANONYMIZATION}
{OOV}
COS
I'VE
NEVER
HEARD
sp
OF
sp
BROWNING
SINCE
sp
THAT
PAPER
sp
SO
sp
WHAT
sp
I
sp
WANT
TO
DO
sp
IS
TO
sp
INTRODUCE
THE
BASIC
IDEA
OF
WHAT
sp
THE
sp
LEARNING
{OOV}
DOES
sp
AND
HOW
THE
NETWORK
sp
SORT
OF
WORKS
sp
{OOV}
sp
UNDERNEATH
IT
sp
AND
THEN
SHOW
HOW
YOU
sp
REPRESENT
IT
AS
sp
A
sp
NEURAL
sp
NETWORK
AND
sp
THE
VARIOUS
FLAVOURS
OF
IT
sp
STRAIGHT
OFF
sp
WHY
IT'S
sp
MORE
IMPORTANT
OR
sp
WHY
IT'S
sp
IMPORTANT
TO
TELL
YOU
ABOUT
IT
IS
sp
BECAUSE
IT'S
A
VERY
FAST
NETWORK
ALL
THE
OTHER
NETWORKS
WE
SEE
sp
I
sp
THINK
sp
THE
NETWORKS
WE'VE
SEEN
NOT
sp
MANY
OF
sp
THEM
sp
ARE
sp
SLOW
IN
LEARNING
sp
AND
THEY'RE
sp
SLOW
IN
LEARNING
BECAUSE
THEY
OPTIMIZE
sp
ER
sp
USING
THIS
LEARNING
RULE
sp
EXACTLY
WHERE
THE
SURFACE
HAS
TO
sp
BE
sp
THE
N
TUPLE
METHOD
TAKES
sp
A
TOTALLY
DIFFERENT
APPROACH
AS
YOU
sp
WILL
SEE
SO
sp
HOW'S
IT
WORK
WELL
IT'S
EXTREMELY
SIMPLE
sp
AND
IF
YOU'RE
ANY
GOOD
AT
DIGITAL
ELECTRONICS
IT
sp
SHOULD
sp
BE
AN
sp
ABSOLUTE
DODDLE
sp
FOR
YOU
COS
IN
FACT
sp
IT'S
ERM
sp
CAN
BE
SEEN
sp
ER
sp
AS
sp
CREATING
sp
FUNCTIONS
IN
DIGITAL
CIRCUITS
sp
IN
sp
VERY
SIMPLE
sp
TERMS
sp
ONE
OF
THE
REASONS
FOR
ITS
SPEED
COMES
sp
FROM
THAT
sp
SO
WE'VE
GOT
sp
SOME
EXAMPLES
HERE
OF
SOME
sp
PATTERNS
THAT
sp
WE
WANT
TO
LEARN
sp
USING
THE
N
TUPLE
METHOD
sp
AND
sp
TUPLE
AND
sp
TUPLE
sp
ERM
sp
EXAMPLE
THERE
sp
OF
T
YOU'LL
sp
RECOGNIZE
THE
sp
T
NOW
sp
THE
sp
FIRST
THING
I
SHOULD
sp
REMIND
YOU
IS
OF
sp
COURSE
sp
THAT
sp
TO
RECOGNIZE
ANYTHING
sp
WE
MUST
HAVE
SOMETHING
sp
TO
COMPARE
IT
AGAINST
sp
OKAY
TO
SAY
HOW
SIMILAR
IT
IS
TO
ONE
sp
THING
TO
ANOTHER
I'VE
TALKED
sp
TO
YOU
sp
ABOUT
THAT
sp
BEFORE
sp
BUT
ANYWAY
WE'RE
JUST
PICKING
ONE
PARTICULAR
EXAMPLE
HERE
AND
sp
WE'VE
sp
GOT
A
T
sp
AND
IT'S
A
VERY
SMALL
sp
ER
sp
EXAMPLE
sp
OF
A
sp
PATTERN
IT'S
sp
GOT
sp
THREE
BYTERY
sp
PIXELS
sp
AND
THE
FIRST
THING
YOU
NOTICE
IS
sp
THE
PIXELS
ARE
BINARY
sp
THEY'RE
NOT
CONTINUOUS
sp
AND
sp
IN
THE
N
TUPLE
METHOD
AS
sp
IT'S
sp
FIRST
sp
DEFINED
sp
THE
ER
sp
PIXELS
WERE
ONLY
BINARY
sp
IT'S
CALLED
THE
sp
N
TUPLE
METHOD
IS
BECAUSE
THE
sp
FIRST
THING
sp
WE
DO
IS
TO
TAKE
THAT
IMAGE
AND
sp
BREAK
IT
UP
INTO
A
SET
OF
sp
TUPLES
sp
EACH
OF
sp
N
IN
SIZE
sp
OKAY
AND
sp
WHAT
WE'VE
GOT
sp
N
HERE
sp
WE'VE
SELECTED
AS
THREE
sp
AND
SO
WE'VE
sp
BROKEN
IT
UP
INTO
THREE
TUPLES
sp
BECAUSE
sp
ANOTHER
RULE
IN
THE
METHOD
EVEN
THOUGH
IT
CAN
BE
BROKEN
sp
IS
THAT
EACH
PIXEL
CAN
ONLY
GO
INTO
sp
ONE
sp
TUPLE
sp
SO
WE'VE
GOT
THE
FIRST
ONE
HERE
F
sp
ONE
sp
SECOND
ONE
HERE
F
sp
TWO
sp
AND
sp
THE
THIRD
ONE
sp
F
sp
THREE
sp
OKAY
sp
ONCE
YOU'VE
DONE
THAT
sp
WHAT
WE
ACTUALLY
DO
IS
TO
LOOK
sp
FOR
sp
FOR
sp
SUB
PATTERNS
sp
IN
sp
THOSE
TUPLES
OF
sp
COURSE
WE'VE
GOT
THE
TRAINING
AND
TESTING
THE
SAME
AS
IN
sp
NORMAL
NETWORKS
sp
THE
TRAINING
sp
IS
sp
OF
sp
RECORDING
sp
WHICH
SUB
PATTERNS
OCCUR
sp
OR
WHICH
PATTERNS
OCCUR
sp
WITHIN
THE
TUPLE
sp
TESTING
sp
CONSISTS
sp
OF
sp
IDENTIFYING
sp
WHICH
OF
THE
KNOWN
ONES
sp
WHICH
sp
OF
sp
THE
KNOWN
ER
SUB
PATTERNS
sp
ER
OCCUR
IN
MY
NEW
INPUT
LET'S
JUST
SEE
AN
EXAMPLE
OF
THAT
F
ONE
sp
HERE
sp
IS
sp
THIS
SUB
PATTERN
HERE
FOR
sp
THIS
sp
TRAINING
EXAMPLE
sp
WE'VE
GOT
NOT
sp
X
sp
ONE
NOT
X
TWO
NOT
X
THREE
AND
I
THINK
ON
YOUR
sp
EXAMPLES
YOU'VE
ACTUALLY
GOT
THEM
sp
LABELLED
sp
ERM
sp
WHICH
MEANS
THAT
sp
EACH
sp
ONE
OF
THOSE
IS
OFF
sp
THE
LOGICAL
sp
ER
sp
FUNCTION
sp
THERE
sp
DETERMINATELY
SIMPLE
sp
F
TWO
THE
SECOND
TUPLE
sp
X
FOUR
IS
ON
sp
X
sp
FIVE
IS
OFF
sp
X
SIX
IS
ON
sp
AND
THE
sp
SAME
SORT
OF
THING
sp
FOR
F
sp
THREE
THERE
sp
SO
WHAT
WE'VE
DONE
IS
WE'VE
sp
APPLIED
sp
A
SIMPLE
LEARNING
RULE
WHICH
JUST
SAYS
sp
REMEMBER
THE
SUB
PATTERNS
sp
TERRIBLY
TRIVIAL
sp
AND
YOU
sp
WONDER
HOW
IT
WOULD
EVER
BE
ANY
GOOD
sp
AT
ANYTHING
sp
SO
THAT'S
WHAT
I
HOPE
sp
TO
SHOW
YOU
sp
ONCE
YOU'VE
sp
TRAINED
IT
LIKE
THAT
sp
TO
TEST
IT
ON
THIS
PATTERN
sp
ALL
YOU
DO
IS
sp
TAKE
THE
PACK
INTO
THE
SYSTEM
AND
sp
SEE
WHETHER
YOU
CAN
sp
{OOV}
THESE
TUPLES
AGAIN
sp
SO
HERE
WE'D
ACTUALLY
HAVE
A
MATCH
OF
sp
ALL
THREE
sp
SAYING
THAT
WE'VE
sp
RECOGNIZED
IT
sp
TO
A
LEVEL
sp
THREE
sp
OKAY
sp
ERM
ON
sp
THAT
PARTICULAR
PATTERN
sp
NOW
AS
IT
STANDS
THAT'S
ALL
IT
KNOWS
sp
AND
sp
IF
sp
IF
THIS
WAS
sp
COVERED
UP
HERE
sp
I
E
THIS
sp
THIS
sp
BLACK
DOT
WAS
SAY
MADE
WHITE
sp
LET'S
SAY
sp
WE'D
ONLY
HAVE
sp
TWO
MATCHING
AND
IF
sp
THAT
WAS
sp
MADE
WHITE
WE'D
HAVE
ONE
AND
THEN
sp
IF
WE
CHANGED
AT
ALL
THAT
sp
TOP
ROW
WE'D
HAVE
ZERO
sp
MATCHING
sp
SO
WHAT
WE
CAN
sp
WHAT
IT
EFFECTIVELY
DOES
sp
IS
sp
COUNT
SORT
OF
SUB
PATTERNS
sp
WITHIN
THE
WHOLE
PATTERN
AND
HOW
WELL
WE'RE
RECOGNIZING
sp
THOSE
sp
IT'S
sp
RATHER
LIKE
WHAT
WE
TALKED
ABOUT
RIGHT
AT
THE
BEGINNING
sp
WE
TALKED
ABOUT
sp
PICKING
OUT
FEATURES
FROM
OUR
IMAGE
sp
SUCH
AS
sp
THE
sp
THE
WIDTH
sp
THE
HEIGHT
ER
sp
THE
sp
GREY
AREA
OR
WHATEVER
sp
OF
sp
THE
IMAGE
sp
NOW
IN
THAT
CASE
sp
WE
sp
HAD
TO
SELECT
sp
THOSE
sp
MEASURES
sp
ER
BY
sp
KNOWLEDGE
OF
THE
sp
PROBLEM
sp
A
TUPLE
IS
A
VERY
SIMILAR
SORT
OF
THING
sp
IT
IT'S
A
sp
IT'S
A
MEASUREMENT
AREA
sp
HERE
sp
FOR
EXAMPLE
sp
BUT
ALL
IT
DOES
IS
sp
RECORD
sp
THE
BINARY
PATTERN
sp
ER
UNDER
THAT
AREA
sp
WE'VE
ACTUALLY
TAKEN
THREE
MEASUREMENT
sp
AREAS
sp
ONE
sp
HERE
sp
ONE
IN
THE
MIDDLE
sp
ONE
AT
THE
BOTTOM
sp
THE
IMPORTANT
THING
IS
THAT
THEY'RE
VERY
ARBITRARY
sp
YOU
sp
YOU
DON'T
HAVE
TO
sp
KNOW
sp
ANYTHING
ABOUT
sp
THE
sp
PROBLEM
sp
TO
SPECIFY
THEM
sp
WHERE
WE'RE
TAKING
sp
MEASURES
FROM
THE
IMAGE
sp
ER
SUCH
AS
sp
HEIGHT
WIDTH
ETCETERA
sp
WE'D
HAVE
TO
KNOW
A
LOT
ABOUT
THE
IMAGE
TO
sp
TAKE
THOSE
sp
MEASURES
sp
THE
IMPORTANT
THING
ABOUT
THE
N
TUPLE
METHOD
IS
THAT
YOU
DON'T
sp
MAKE
ANY
ASSUMPTIONS
ABOUT
THE
IMAGE
APART
FROM
IT'S
JUST
A
SET
OF
BINARY
DIGITS
sp
OKAY
sp
WHICH
IS
sp
ONE
OF
THE
sp
THE
STRENGTHS
OF
sp
COURSE
sp
FOR
THAT
SORT
OF
THING
sp
SO
WHAT
WE'RE
DOING
EFFECTIVELY
IS
sp
SEEING
HOW
MANY
FEATURES
sp
WE
RECOGNIZE
sp
IN
OUR
sp
IN
sp
OUR
ER
sp
EXAMPLE
sp
IMAGE
sp
GENERALIZATION
WHICH
WE'LL
COME
TO
A
LOT
LATER
AS
sp
WELL
IN
A
LOT
MORE
sp
DEPTH
sp
ER
sp
IS
VERY
IMPORTANT
IN
sp
THESE
{OOV}
SYSTEMS
sp
I
MAY
WANT
TO
RECOGNIZE
THIS
sp
PATTERN
sp
AS
WELL
AS
THAT
sp
PATTERN
sp
BUT
I
MAY
WANT
TO
RECOGNIZE
THAT
sp
JUST
AS
WELL
AS
THAT
sp
I
E
AT
THE
MOMENT
THIS
sp
PATTERN
sp
FED
INTO
THESE
FUNCTIONS
HERE
sp
GIVES
US
THREE
MATCHING
FUNCTIONS
sp
IF
WE
ADD
AN
EXTRA
TERM
TO
THE
BOTTOM
OF
HERE
sp
REPRESENTING
THIS
sp
EXTRA
sp
CHANGED
sp
PATTERN
sp
OKAY
sp
THIS
sp
PATTERN
WILL
MATCH
sp
JUST
AS
WELL
AS
sp
THAT
sp
ER
sp
OUR
sp
FUNCTIONS
sp
I
E
sp
THIS
sp
ONE
sp
GIVES
US
A
sp
A
VALUE
OF
THREE
sp
SO
sp
REALLY
THAT'S
sp
IT
AS
FAR
AS
sp
THE
N
TUPLE
METHOD
sp
IS
sp
YOU'RE
JUST
MEASURING
sp
SUB
sp
SUB
PATTERNS
OF
THE
IMAGE
sp
COUNTING
HOW
MANY
OF
THOSE
sp
MATCH
sp
IN
TRAINING
sp
IN
TESTING
sp
AND
THAT
COUNT
sp
IS
A
MEASURE
OF
SIMILARITY
sp
YOU
CAN
THINK
ABOUT
IT
AND
IT'S
VERY
USEFUL
TO
THINK
ABOUT
IT
AS
A
AS
A
sp
DISTANCE
sp
MEASURE
sp
IN
A
sp
SENSE
sp
IT'S
A
VERY
COMPLEX
sp
DISTANCE
sp
MEASURE
sp
COMPARED
TO
sp
THE
sp
ONES
sp
WE'VE
sp
LOOKED
sp
AT
sp
BUT
IT
IS
A
DISTANCE
sp
MEASURE
sp
COS
THIS
WHOLE
FUNCTION
sp
PLUS
THE
SUMMATION
OF
sp
OF
sp
HOW
MANY
sp
TERMS
MATCH
sp
IS
A
MEASURE
OF
HOW
SIMILAR
sp
OUR
sp
UNKNOWN
PATTERN
sp
IS
sp
ER
TO
ALL
OUR
KNOWN
ONES
THAT
sp
WE'VE
TRAINED
THE
SYSTEM
ON
sp
BUT
IT'S
sp
RATHER
sp
STRANGE
AND
sp
ER
CERTAINLY
NOT
sp
{OOV}
DISTANCE
sp
MEASURE
sp
BUT
sp
IT
IS
A
sp
IT
sp
IS
sp
A
sp
MEASURE
sp
AND
OF
COURSE
sp
THE
WHOLE
THING
IS
sp
VERY
sp
EASY
TO
sp
COMPUTE
sp
REALLY
sp
AFTER
THE
N
TUPLE
METHOD
sp
WAS
DEFINED
IN
NINETEEN
sp
FIFTY
NINE
sp
ERM
THE
NEXT
sp
PERSON
sp
WHO
sp
CAME
ALONG
sp
THAT
sp
DID
ANYTHING
sp
WITH
IT
sp
WAS
sp
IGOR
ALEXANDER
{OOV}
sp
SUPERVISOR
sp
THAT'S
sp
WHY
I
WORK
sp
ON
sp
THIS
sp
AREA
sp
GOT
sp
A
LOT
TO
sp
BLAME
sp
TO
sp
BLAME
sp
FOR
I
sp
MUST
sp
ADMIT
sp
WHAT
HE
DID
HE
SAW
sp
HE
SAW
THE
METHOD
AND
SAW
sp
WELL
THIS
sp
IS
sp
QUITE
sp
NEAT
sp
WE
CAN
ACTUALLY
sp
IMPLEMENT
IT
sp
VERY
EFFICIENTLY
IN
HARDWARE
sp
WE
CAN
sp
MADE
sp
A
VERY
sp
FAST
sp
RECOGNITION
sp
SYSTEM
sp
OUT
sp
OF
IT
sp
WHAT
WE'VE
GOT
HERE
sp
IS
sp
A
VERY
CONVENTIONAL
sp
PIECE
OF
LOGIC
THAT
IMPLEMENTS
WHAT
I'VE
TALKED
ABOUT
sp
WE'VE
GOT
TWO
TUPLES
SO
I'VE
sp
GIVEN
THE
sp
EXAMPLE
AGAIN
WE'VE
GOT
TWO
TUPLES
sp
OF
TWO
sp
{OOV}
TWO
sp
ONE
HERE
AND
sp
ONE
HERE
WE'D
HAVE
OTHER
ONES
AS
WELL
BUT
sp
JUST
TO
SHOW
THIS
sp
PARTICULAR
sp
EXAMPLE
JUST
sp
TO
sp
KEEP
IT
sp
SMALL
sp
I'LL
sp
MAKE
sp
IT
sp
SMALL
sp
WE
THEN
HAVE
sp
BINARY
DECODERS
sp
AND
AS
YOU
KNOW
BINARY
DECODERS
LIST
sp
ALL
THE
POSSIBLE
sp
STATES
sp
OF
sp
THE
sp
INPUT
sp
ERM
sp
CONDITIONS
sp
SO
HERE
WE'VE
GOT
BINARY
INPUTS
TO
THERE
MUST
BE
TWO
TO
THE
sp
TWO
sp
OUTPUTS
sp
I
E
sp
FOUR
POSSIBLE
sp
COMBINATIONS
sp
AND
HERE'S
THE
TRUTH
TABLE
FOR
THIS
PARTICULAR
DECODER
sp
OKAY
AND
THESE
YOU
SHOULD
REMEMBER
sp
ARE
BASICALLY
THE
TERMS
I
GAVE
ON
THE
sp
PREVIOUS
sp
PAGE
sp
SAME
AS
THESE
TERMS
HERE
sp
EXCEPT
sp
SO
THE
DECODERS
ARE
ACTUALLY
LISTING
ALL
THE
POSSIBLE
TERMS
OF
sp
THE
INPUTS
sp
THE
sp
MEMORY
CELLS
HERE
OF
sp
COURSE
sp
RECALLED
WHETHER
THAT
TERM
sp
OCCURRED
sp
DURING
sp
TRAINING
sp
SO
I
PRESENT
THIS
sp
PATTERN
sp
I
PUSH
IT
THROUGH
THE
DECODERS
sp
AND
I
SEE
THAT
THAT
LINE'S
sp
MAYBE
IS
ACTIVE
SO
I
PUT
A
ONE
IN
THERE
TO
RECORD
THE
sp
FACT
THAT
THAT
LOGIC
sp
TERM
ER
sp
OCCURRED
AND
sp
SIMILARLY
FOR
sp
THIS
sp
ER
DECODER
DOWN
HERE
THIS
SECTION
sp
DOWN
HERE
sp
SO
TRAINING
IS
VERY
EASY
TO
IMPLEMENT
LIKE
sp
THAT
WITH
A
FEW
MEMORY
CELLS
AND
A
BINARY
DECODER
sp
RECALL
IS
SIMILARLY
VERY
SIMPLE
YOU
PRESENT
THE
INPUT
sp
PATTERN
sp
PUSHING
THROUGH
THE
DECODERS
AND
INSTEAD
OF
READING
sp
FROM
THESE
sp
MEMORY
LOCATIONS
sp
SORRY
WRITING
sp
FROM
THESE
MEMORY
LOCATIONS
sp
YOU
READ
FROM
THEM
sp
OKAY
SO
IN
THIS
CASE
YOU
GET
{OOV}
THIS
sp
R
WOULD
{OOV}
sp
BE
ACTIVATED
AND
WE
ACCESS
THIS
sp
LOCATION
HERE
sp
WHICH
WOULD
BE
SUMMED
sp
INTO
SUMMATION
TO
GIVE
US
OUR
RESPONSE
sp
OF
sp
WELL
TWO
sp
MAYBE
IN
THIS
sp
PARTICULAR
sp
CASE
sp
AND
IF
IT
ACTUALLY
RECOGNIZED
sp
IT
OKAY
SO
WE
CAN
IMPLEMENT
IT
LIKE
THAT
IN
HARDWARE
BUT
THE
sp
MAJOR
BENEFIT
sp
AND
THIS
WAS
sp
DONE
SORT
OF
LIKE
sp
SIXTIES
sp
SORT
sp
OF
SIXTY
sp
NINE
SEVENTY
sp
WAS
TO
REALIZE
THAT
YOU
sp
COULD
sp
PUT
IT
INTO
A
sp
LITTLE
PACKAGE
LIKE
THAT
WHICH
OF
sp
COURSE
sp
IS
A
RANDOM
ACCESS
sp
MEMORY
sp
THE
TUPLE
LINES
sp
ARE
sp
THE
ADDRESS
INPUTS
INTO
sp
YOUR
sp
RANDOM
ACCESS
sp
MEMORY
sp
THE
MEMORY
IS
JUST
THE
MEMORY
THE
DATA
INPUT
sp
STORES
THE
ONE
sp
FOR
THE
ADDRESS
sp
LOCATION
AND
THE
DATA
OUTPUT
STORES
sp
THE
OUTPUT
sp
AND
YOU'VE
GOT
TA
REWRITE
SIGNAL
sp
TO
sp
TELL
YOU
WHETHER
IT'S
sp
GOT
TA
LEARN
OR
sp
TO
sp
RECOGNIZE
THE
IMAGE
sp
NOW
HERE'S
AN
INTERESTING
STORY
ABOUT
THIS
sp
ER
sp
IGOR
ALEXANDER
WHO
sp
WHO
sp
AS
I
SAY
sp
RECOGNIZES
STARTED
TO
BUILD
THESE
sp
THESE
LITTLE
sp
CELLS
sp
THESE
THINGS
sp
BEFORE
sp
RANDOM
ACCESS
sp
MEMORIES
CAME
ALONG
sp
HE
CALLED
THEM
sp
ERM
WHAT
sp
WERE
THEY
{OOV}
sp
SOMETHING
IT
sp
WAS
sp
SOMETHING
sp
STRANGE
LIKE
sp
THAT
sp
BUT
THE
sp
POINT
WAS
THAT
sp
RANDOM
ACCESS
sp
MEMORIES
NEVER
EXISTED
sp
WHEN
sp
HE
FIRST
sp
DID
THIS
AND
sp
HE
WENT
TO
sp
PLESSEY
AND
sp
ASKED
PLESSEY
IF
sp
HE
COULD
IMPLEMENT
sp
THESE
LITTLE
sp
FUNCTIONS
sp
ON
MICRO
CIRCUITS
sp
I
E
SORT
sp
OF
ER
sp
THE
L
S
I
TYPE
CIRCUITS
sp
AND
HE
GOT
SOME
PACKAGES
sp
MADE
BUT
UNFORTUNATELY
HE
DIDN'T
PATENT
sp
THE
IDEA
sp
{LG}
sp
BECAUSE
WHAT
HE
ACTUALLY
WAS
MADE
sp
WHAT
HE
ACTUALLY
MADE
WAS
RANDOM
ACCESS
sp
MEMORIES
sp
ERM
AND
HAD
HE
sp
HAD
HE
PATENTED
IT
OR
sp
I
DON'T
KNOW
ABOUT
EXACTLY
THE
TIMING
HE
MIGHT
HAVE
MADE
HIMSELF
AN
ABSOLUTE
sp
FORTUNE
NOT
THAT
HE
sp
WAS
INTERESTED
sp
PARTICULARLY
sp
BUT
ER
sp
IT'S
AN
INTERESTING
STORY
sp
ERM
sp
HE
DID
THAT
WHEN
HE
WAS
AT
sp
ERM
sp
ER
sp
NOW
sp
WHERE
WAS
IT
sp
KENT
sp
UNIVERSITY
OF
KENT
sp
JUST
sp
THE
STRANGE
THINGS
THAT
AGAIN
YOU
KNOW
sp
IT'S
sp
IT'S
HAVING
AN
IDEA
AND
RECOGNIZING
IT
THAT
IT
WOULD
BE
USEFUL
sp
IT'S
SOMETHING
sp
THAT
ER
sp
NOT
ALWAYS
HAPPENS
sp
LIKE
sp
IN
sp
THE
BACK
PROPAGATION
sp
LEARNING
sp
{OOV}
AND
sp
PEOPLE
sp
WHO
sp
INVENTED
sp
IT
sp
DIDN'T
REALIZE
sp
IT
sp
WAS
THAT
sp
THAT
IMPORTANT
sp
AND
DIDN'T
ADVERTISE
IT
sp
HE
INVENTED
sp
THE
sp
RANDOM
ACCESS
sp
MEMORY
sp
BUT
sp
ER
DIDN'T
sp
TELL
ANYBODY
ABOUT
sp
IT
sp
NEVER
sp
MIND
sp
OKAY
SO
WE
CAN
IMPLEMENT
THIS
SORT
OF
THING
sp
IN
sp
ER
sp
USING
RANDOM
ACCESS
sp
MEMORIES
sp
AND
AS
sp
SUCH
OF
sp
COURSE
sp
IN
sp
RUN
LOAD
sp
YOU
CAN
sp
MAKE
THIS
THING
GO
IN
THE
ORDER
OF
TENS
OF
NANOSECONDS
sp
PROBABLY
LESS
sp
TO
RECOGNIZE
AN
IMAGE
sp
BUT
AS
I
SAY
EVERY
YEAR
ER
IF
sp
I
sp
EVER
FIND
AN
APPLICATION
WHERE
YOU
NEED
THIS
SORT
OF
SPEED
sp
I'D
LIKE
TO
HEAR
ABOUT
IT
sp
ERM
sp
THERE'S
VERY
FEW
APPLICATIONS
THAT
sp
NEED
TO
sp
RECOGNIZE
ANYTHING
sp
IN
THAT
sp
AMOUNT
sp
OF
TIME
sp
THE
ONLY
sp
SMALL
IDEA
THAT
I
HAVE
AND
I
DON'T
THINK
IT'S
PARTICULARLY
PRACTICAL
sp
IS
sp
FOR
A
SORT
OF
OSCILLOSCOPE
TRIGGERING
MECHANISM
sp
OFTEN
sp
WHICH
IS
IN
FACT
ONE
YOU
GET
IN
LOGIC
ANALYSERS
ANYWAY
sp
BUT
ER
OFTEN
YOU
MIGHT
WANT
sp
TO
TRIGGER
OFF
A
sp
PARTICULAR
sp
PATTERN
sp
COMING
ON
THE
TRACE
sp
AND
YOU
MIGHT
HAVE
TO
sp
YOU
HAVE
TO
DO
THAT
IN
IN
sp
THE
ORDER
OF
NANOSECONDS
sp
BUT
sp
OF
COURSE
sp
SOMETHING
LIKE
sp
THIS
sp
WOULD
DO
THAT
BUT
sp
YOU
HAVE
TO
sp
ALLOW
FOR
THE
JITTER
AS
WELL
sp
AND
sp
THIS
SORT
OF
ALGORITHM
sp
MIGHT
sp
MIGHT
sp
BE
THE
sp
WAY
TO
sp
DO
IT
sp
I
THINK
THERE
ARE
A
FEW
OTHERS
sp
BUT
sp
NOT
MANY
sp
SO
IF
YOU
IMPLEMENT
IT
TOTALLY
IN
PARALLEL
YOU
CAN
GO
VERY
FAST
TYPICALLY
WE
DON'T
sp
WE
IMPLEMENT
IT
IN
sp
SOFTWARE
AND
EVEN
IN
sp
SOFTWARE
IT'S
VERY
sp
QUICK
sp
BECAUSE
sp
COMPUTERS
sp
ARE
IDEALLY
SET
UP
TO
sp
IMPLEMENT
THESE
SORT
OF
SYSTEMS
sp
SO
IT'S
GREAT
sp
IT'S
THE
RIGHT
SORT
OF
TECHNOLOGY
WE'VE
GOT
THE
RIGHT
SORT
OF
COMPUTER
TECHNOLOGY
TO
IMPLEMENT
sp
THESE
sp
ERM
sp
WE
CAN
MOVE
OUT
FROM
THERE
sp
INTO
sp
THE
NEURAL
NETWORK
sp
WORLD
sp
AND
OF
COURSE
sp
IT'S
VERY
HARD
TO
SEE
sp
THIS
AS
A
NEURAL
NETWORK
SO
WE
COULD
ASK
OURSELVES
sp
WELL
VERY
NICE
sp
BUT
WHERE
DOES
NEURAL
NETWORKS
FIT
IN
sp
FACT
RECOGNITION
sp
METHOD
IT'S
sp
A
sp
PIECE
OF
HARDWARE
AND
sp
ONE
OF
THE
THINGS
THAT'S
sp
CLEAR
ACTUALLY
IS
THAT
ER
sp
THE
ALEXANDER
sp
WORLDS
AND
ALL
SORTS
sp
OF
OTHER
PEOPLE
THAT
WERE
WORKING
sp
ON
THIS
sp
DON'T
sp
LOOK
AT
THEM
IN
THIS
WAY
sp
AND
sp
THEY
LOOK
AT
THESE
AS
sp
I
SAY
RAM
sp
BLOCKS
sp
ERM
sp
AS
A
RESULT
THEY
CALL
THEM
WEIGHTLESS
sp
NETWORKS
sp
WHICH
sp
WE
ACTUALLY
sp
HATE
BECAUSE
sp
THEY'RE
NOT
sp
WEIGHTLESS
IF
sp
YOU
ACTUALLY
REPRESENT
THEM
sp
LIKE
sp
I'VE
JUST
SHOWN
YOU
sp
YOU
CAN
CHANGE
THE
DIAGRAM
VERY
SLIGHTLY
AND
THEY
LOOK
sp
JUST
LIKE
sp
THE
NORMAL
NEURAL
NETWORK
IN
sp
FACT
sp
IT'S
VERY
MUCH
LIKE
sp
THE
sp
HIGHER
ORDER
NETWORKS
sp
THAT
I
TALKED
sp
ABOUT
sp
IN
THE
LAST
LECTURE
ER
sp
I
sp
THINK
IT
sp
WAS
sp
THE
sp
LAST
LECTURE
BUT
sp
ONE
sp
AND
THE
NETWORKS
sp
WHICH
sp
PRE
sp
PROCESS
THE
sp
DATA
sp
BEFORE
THEY'RE
PRESENTED
TO
sp
THE
NETWORK
sp
BY
SOME
sp
HIGHER
{OOV}
FUNCTION
sp
WHAT
YOU
HAVE
HERE
sp
IS
sp
BASICALLY
A
SINGLE
NEURON
WITH
A
sp
OF
WEIGHTS
sp
THE
sp
WEIGHTS
sp
ON
THE
CIRCLES
HERE
ARE
sp
THE
BINARY
sp
ELEMENTS
IN
THE
RANDOM
ACCESS
sp
MEMORY
sp
THE
WEIGHT
sp
YOU
KNOW
THE
STORAGE
LOCATIONS
sp
THE
DECODERS
HERE
sp
ARE
JUST
sp
NON
LINEAR
FUNCTIONS
sp
THEY'RE
JUST
sp
BINARY
NON
LINEAR
FUNCTIONS
sp
THEY
sp
HAPPEN
sp
TO
BE
sp
BUT
THEY'RE
NON
LINEAR
sp
NOW
IF
WE
RECOGNIZE
IT
sp
AS
sp
THAT
WE
IMMEDIATELY
SEE
sp
THAT
THE
METHOD
sp
IS
sp
WELL
WHY
THE
METHOD
sp
WORKS
sp
IN
NEURAL
NETWORK
TERMS
sp
IF
YOU
REMEMBER
A
SINGLE
{OOV}
NETWORK
IS
A
LINEAR
CLASSIFIER
sp
AND
CAN'T
SOLVE
NON
LINEARLY
SOL
sp
SOLUBLE
sp
PROBLEMS
sp
TO
GET
ROUND
THAT
ONE
OF
THE
APPROACHES
THAT
I
TALKED
ABOUT
WAS
TO
sp
PRE
sp
PROCESS
sp
THE
DATA
sp
I
E
OR
FORMALIZE
IT
IN
SOME
sp
WAY
sp
FORCE
IT
sp
THROUGH
SOME
sp
SOME
sp
FUNCTIONS
sp
SO
THAT'S
WHAT
sp
THESE
sp
DECODERS
DO
ER
sp
SUCH
sp
THAT
IT
IS
sp
HOPEFULLY
LINEARLY
SEPARABLE
sp
NOW
ONE
OF
THE
PROBLEMS
sp
OVER
ALL
sp
THAT
sp
IS
TO
SAY
sp
WELL
IS
IT
sp
HOW
sp
CAN
sp
CAN
IT
sp
ALWAYS
SOLVE
THE
sp
PROBLEM
WELL
IT
sp
CAN'T
IT
DOESN'T
sp
ALWAYS
SOLVE
THE
sp
PROBLEM
sp
IN
FACT
RECOGNITION
YOU
DON'T
HAVE
sp
TO
GUARANTEE
THAT
sp
YOU
WILL
SOLVE
THE
PROBLEM
sp
BECAUSE
sp
EVEN
THOUGH
THIS
IS
A
sp
THIS
IS
A
LINEAR
NETWORK
IT
sp
CAN
BE
sp
EFFECTIVELY
SEEN
AS
A
NETWORK
EFFECTIVELY
WITH
sp
A
UNIT
HERE
AND
A
UNIT
HERE
AND
MAYBE
ONE
WEIGHT
sp
IF
IT
WAS
SEEN
LIKE
sp
THAT
sp
YOU
CAN'T
ACTUALLY
SOLVE
sp
A
ERM
A
sp
LINEAR
PROBLEM
sp
WHICH
sp
OCCURS
sp
BETWEEN
THESE
TWO
SETS
OF
UNITS
YOU
sp
WILL
SEE
THAT
IN
A
SECOND
BUT
THERE
ARE
LIMITATIONS
WITH
IT
sp
ERM
WHICH
sp
ARE
OKAY
FOR
FACT
RECOGNITION
BUT
NOT
SO
sp
GOOD
sp
FOR
MORE
sp
SORTS
sp
OF
sp
ANALYSIS
PROBLEMS
sp
BUT
sp
WE'LL
SEE
THAT
IN
sp
A
SECOND
sp
OKAY
SO
YOU
CAN
VIEW
IT
AS
A
sp
AS
A
sp
SINGLE
NEURON
sp
WITH
SOME
sp
ER
DECODERS
{OOV}
sp
THE
sp
OTHER
IMPORTANT
sp
THING
IS
THAT
THE
LEARNING
RULE
sp
IS
NOT
A
sp
GRADED
sp
DESCENT
sp
LEARNING
RULE
sp
OR
A
SEARCHING
LEAR
sp
LEARNING
sp
RULE
AS
sp
WE'VE
sp
SEEN
BEFORE
sp
REMEMBER
WHAT
WE
DO
WITH
THE
NETWORK
NORMALLY
IS
TO
sp
PRESENT
AN
INPUT
sp
GET
AN
OUTPUT
sp
SEE
IF
IT'S
CORRECT
sp
IF
IT
IS
sp
CORRECT
sp
WE
DON'T
DO
ANYTHING
AND
IF
IT
ISN'T
sp
CORRECT
sp
WE
USE
THAT
ERROR
sp
TO
ADJUST
sp
THE
WEIGHTS
sp
NOW
sp
HERE
WE'VE
GOT
BINARY
WEIGHTS
sp
SO
IT'S
EITHER
SETTING
OR
sp
NOT
SETTING
sp
THE
sp
WEIGHTS
sp
AND
IN
THE
N
TUPLE
RULE
ALL
WE
DO
IS
IN
FACT
SET
THE
WEIGHTS
sp
SO
WE
PRESENT
THE
PATTERN
sp
WE
COULD
TEST
IT
TO
SEE
IF
WE
GET
AN
OUTPUT
WE
DON'T
BUT
sp
WE
sp
COULD
sp
DO
sp
THERE'S
NO
NEED
BECAUSE
ALL
WE
HAVE
TO
DO
IN
EFFECT
sp
IS
JUST
sp
SET
sp
THE
WEIGHTS
sp
OF
THE
DECODERS
sp
THAT
ARE
ACTUALLY
sp
COMING
sp
OUT
sp
BUT
sp
THE
DECODERS
sp
THAT
ARE
sp
THE
DECODER
LINES
THAT
sp
ARE
sp
ACTIVE
sp
SOMETIMES
THAT
LEARNING
sp
METHOD
IS
CALLED
THE
HEBEAN
sp
LEARNING
RULE
ERM
sp
BECAUSE
sp
IT'S
THE
SORT
OF
THING
YOU
sp
FIND
IN
ANIMALS
sp
ERM
sp
DONALD
sp
{GAP_ANONYMIZATION}
sp
A
FAMOUS
sp
WHAT
sp
WAS
sp
HE
A
FAMOUS
sp
NEUROPHYSIOLOGIST
sp
HE
CAME
UP
sp
WITH
THIS
sp
HOW
sp
NEURONS
sp
ACTUALLY
DID
LEARN
sp
BY
THE
WAY
NEURON
I
HAVEN'T
SAID
THAT
sp
BUT
sp
THE
THE
{OOV}
LEARNING
RULE
IS
N
sp
IS
sp
NOT
sp
BIOLOGICALLY
PLAUSIBLE
sp
AND
A
LOT
OF
LEARNING
RULES
sp
AREN'T
BIOLOGICALLY
sp
PLAUSIBLE
sp
ERM
sp
THIS
sp
HAPPENS
TO
sp
BE
AS
sp
WELL
sp
{OOV}
sp
AS
sp
AN
ASIDE
sp
FOR
A
MULTI
CLASS
PROBLEM
sp
WHERE
YOU'VE
GOT
sp
MORE
THAN
sp
ONE
NEURON
sp
MORE
sp
THAN
ONE
CLASS
sp
YOU
MIGHT
sp
DISPLAY
THE
SYSTEM
LIKE
sp
THIS
sp
IT'S
A
SLIGHTLY
DIFFERENT
WAY
OF
sp
LOOKING
AT
NEURON
NETWORKS
THAN
sp
PERHAPS
WE'VE
SEEN
BEFORE
BECAUSE
WE
HAVE
THIS
sp
MATRIX
HERE
BUT
WHAT
WE'VE
EFFECTIVELY
GOT
IS
A
NEURON
HERE
A
sp
NEURON
HERE
AND
A
NEURON
HERE
AND
sp
WE'VE
GOT
sp
OUR
THREE
CLASSES
sp
AND
WE
sp
PRE
PROCESS
THE
DATA
sp
AS
I
SAY
COMMONLY
FOR
ALL
sp
THE
NEURONS
sp
BEFORE
WE
sp
PRESENT
sp
IT
sp
TO
OUR
sp
THREE
NEURONS
sp
TESTING
FOR
THESE
SORT
OF
SYSTEMS
ERM
sp
WE
sp
HAD
sp
TO
sp
DECIDE
sp
WHICH
OF
THESE
sp
RESPONSES
IS
sp
BEST
sp
WELL
sp
AS
YOU
sp
SEE
WE
CAN
sp
COUNT
THE
NUMBER
OF
TERMS
THAT
ACTUALLY
MATCH
sp
FROM
sp
OUR
sp
TRAINING
SETS
SO
sp
THIS
EXAMPLE
HERE
WE
MIGHT
sp
FIND
THAT
THAT
sp
MATCHES
THAT
sp
MATCHES
sp
WE
GET
A
RESPONSE
OF
sp
TWO
sp
FROM
HERE
sp
AND
sp
WE
GET
A
RESPONSE
sp
LESS
THAN
THE
RESPONSE
FROM
HERE
sp
AND
THE
SAME
RESPONSE
sp
FROM
THIS
sp
WE
sp
HAVE
TO
MAKE
A
DECISION
ON
THOSE
RESPONSES
JUST
LIKE
IN
A
sp
NORMAL
NETWORK
NOW
WE
COULD
JUST
TAKE
THE
PATTERN
OF
RESPONSES
OUT
OF
sp
THERE
AS
BEING
SOME
sp
{OOV}
sp
CLASSES
YOU
CAN
DO
WHAT
YOU
LIKE
sp
REALLY
ERM
sp
BUT
sp
THE
EASIEST
ONE
IS
OBVIOUSLY
TO
TAKE
THE
MAXIMUM
RESPONDING
OUTPUT
sp
FROM
sp
HERE
sp
AS
sp
ER
sp
AN
INDICATION
OF
WHICH
CLASS
IS
sp
BELONGS
sp
TO
sp
OKAY
sp
SO
THAT'S
ER
A
sp
VERY
STRAIGHTFORWARD
sp
SORT
sp
OF
THING
sp
THE
OTHER
THING
YOU
SHOULD
REMEMBER
IS
THAT
THERE
ARE
ALL
SORTS
OF
CONFIDENCES
THAT
YOU
CAN
APPLY
TO
sp
THESE
SORT
sp
OF
THINGS
sp
THAT
IF
WE
HAD
A
RESPONSE
HERE
OF
sp
TWO
ONE
ZEROS
I'VE
sp
SCORED
AT
THE
sp
BOTTOM
OF
sp
THAT
sp
WE'VE
ACTUALLY
GOT
sp
TWO
OF
THE
LOGIC
TERMS
MATCHING
FROM
THIS
PARTICULAR
sp
CLASS
sp
ONE
OF
THE
LOGIC
TERMS
MATCHING
FROM
sp
THIS
PARTICULAR
CLASS
AND
sp
NONE
FOR
THIS
sp
PARTICULAR
sp
CLASS
sp
HOW
WELL
HOW
sp
HOW
sp
SURE
ARE
WE
OF
THIS
RESULT
HERE
WELL
WE'RE
sp
TOO
sp
SURE
sp
AREN'T
WE
WE'RE
sp
SAYING
THAT
sp
WE'VE
GOT
sp
TWO
OF
OUR
sp
OUR
sp
PARTICULAR
SUB
PATTERNS
MATCHING
sp
COS
WE'VE
GOT
A
RESPONSE
OF
TWO
sp
HERE
WE'VE
GOT
ONE
SO
WE
MIGHT
SAY
WELL
OUR
OVERALL
CONFIDENCE
IS
TWO
sp
BUT
OUR
RELATIVE
sp
CONFIDENCE
sp
WHICH
IS
SAYING
HOW
DIFFERENT
IT
sp
IS
FROM
EVERYTHING
ELSE
sp
IS
ONLY
ONE
sp
AND
YOU
ACTUALLY
sp
DO
GET
sp
TWO
TYPES
OF
sp
OF
sp
OF
sp
THRESHOLD
ERM
sp
CONFIDENCE
IN
THESE
SORT
OF
SYSTEMS
sp
THERE'S
THIS
sp
ABSOLUTE
sp
CONFIDENCE
sp
WHICH
IS
sp
THE
ABSOLUTE
OUTPUT
OF
sp
MAXIMUM
RESPONDING
NEURON
sp
THEN
THERE'S
THE
RELATIVE
sp
CONFIDENCE
sp
WHICH
IS
TO
SAY
HOW
sp
MUCH
sp
STRONGER
IS
THAT
sp
PARTICULAR
OUTPUT
sp
COMPARED
TO
ANY
OTHERS
sp
AND
BOTH
OF
THOSE
FACTORS
sp
CAN
BE
IMPORTANT
IN
sp
JUDGING
HOW
THE
SYSTEM'S
sp
DOING
sp
OKAY
sp
THAT
BASIC
ARCHITECTURE
IS
IF
sp
YOU
WANT
THE
VANILLA
sp
SORT
OF
ARCHITECTURE
FOR
THE
ENTRICAL
sp
STUFF
sp
ERM
sp
THERE
ARE
sp
A
LOT
OF
VARIATIONS
sp
UPON
IT
sp
WHICH
sp
ER
MAKE
IT
INTERESTING
AND
WE'LL
SEE
LATER
ALSO
HOW
sp
THE
THING
ACTUALLY
DOES
sp
WORK
AND
sp
ALL
ITS
sp
PROPERTIES
sp
THE
VARIATIONS
sp
TO
THE
SYSTEM
COME
IN
A
NUMBER
OF
FORMS
FIRST
OF
ALL
sp
THAT
THE
INPUT
sp
CAN
sp
BE
GREY
SCALE
OF
sp
COURSE
sp
INSTEAD
OF
BINARY
sp
SO
sp
HOW
DOES
THE
SYSTEM
sp
COPE
WITH
THAT
WELL
WE'VE
ONLY
GOT
BINARY
DECODERS
HERE
sp
SO
WE
HAVE
TO
sp
COPE
sp
WE
HAVE
TO
sp
CHANGE
SOMETHING
sp
MAYBE
WE
CHANGE
THE
DECODERS
sp
MAYBE
WE
CHANGE
THE
INPUTS
sp
THERE'S
A
WHOLE
sp
SECTION
OF
HOW
WE
DO
THAT
sp
THESE
sp
THIS
ARRAY
sp
OF
WEIGHTS
IS
BINARY
sp
WELL
WE
COULD
HAVE
CONTINUOUS
sp
WEIGHTS
IN
sp
THAT
sp
PARTICULAR
ARRAY
sp
POSSIBLY
sp
WHAT
WHAT
BENEFIT
sp
WOULD
THAT
DO
sp
FOR
US
sp
OF
COURSE
IT
IT
SCREWS
UP
THE
IMPLEMENTATION
ER
IF
YOU
sp
WANT
sp
TO
IMPLEMENT
IN
sp
HARDWARE
AND
THESE
AREN'T
BINARY
WEIGHTS
IT
sp
STARTS
sp
TO
MAKE
sp
THE
sp
THE
sp
CIRCUITRY
A
BIT
MORE
sp
COMPLICATED
sp
THE
THIRD
THING
IS
sp
IN
THIS
SUMMATION
sp
FUNCTION
sp
WHY
DO
WE
USE
A
SUMMATION
sp
FUNCTION
I
MEAN
IN
THE
NEURAL
NETWORK
sp
TERMS
sp
WE
JUST
USE
A
SUMMATION
sp
FUNCTION
BUT
WE
SHOULD
ASK
OURSELVES
sp
WHY
ARE
WE
USING
sp
THAT
sp
AND
WHAT'S
THE
EFFECT
OF
USING
OTHER
FUNCTIONS
IN
THERE
SUCH
AS
MAYBE
AN
AND
OR
AN
OR
sp
WHERE
WE
sp
OR
sp
THE
RESPONSES
OUT
OF
HERE
OR
WE
AN
THEM
BECAUSE
NOW
WE'VE
GOT
A
BINARY
NETWORK
WE
CAN
DO
DIFFERENT
THINGS
sp
IT'S
THE
sp
EQUIVALENT
TO
IN
A
NEURAL
NETWORK
SAYING
sp
INSTEAD
OF
SAYING
THE
WEIGHTS
TIMES
THE
INPUTS
ALL
SUMMED
sp
WE'LL
TAKE
EACH
WEIGHT
TIMES
ITS
INPUT
AND
sp
MULTIPLY
IT
WITH
EACH
sp
WEIGHT
sp
TIMES
ITS
INPUT
sp
SO
WE
DO
THE
PRODUCT
sp
OF
ALL
THE
sp
INPUTS
sp
TIMES
sp
THE
sp
WEIGHTS
sp
ERM
sp
WHAT
sp
EFFECT
WOULD
THAT
HAVE
AND
sp
THAT'S
INTERESTING
TO
SEE
IN
PATTERN
sp
RECOGNITION
TERMS
sp
REALLY
THAT
BRINGS
OUT
ONE
OF
THE
BENEFITS
OF
sp
OF
USING
THIS
SORT
OF
NETWORK
sp
BECAUSE
YOU
CAN
ANALYSE
sp
IT
sp
YOU
CAN
LOOK
sp
RIGHT
INTO
IT
AND
YOU
CAN
ACTUALLY
sp
FIND
OUT
sp
THE
BEHAVIOUR
OF
IT
sp
FULLY
YOU
sp
CAN
SEE
HOW
WELL
IT
GENERALIZES
YOU
CAN
sp
SEE
sp
HOW
WELL
sp
YOU
sp
CAN
sp
IMPLEMENT
IT
sp
WHICH
sp
IS
sp
NICE
OKAY
I
sp
THINK
YOU'VE
GOT
A
CORRECTED
sp
SLIDE
sp
HERE
ERM
sp
THIS
IS
sp
MY
ATTEMPT
AT
A
FORMAL
DEFINITION
sp
OF
sp
THE
NETWORK
sp
ERM
sp
IT
HELPS
YOU
IF
YOU
WANT
TO
WORK
OUT
sp
EXACTLY
HOW
IT
WORKS
AND
sp
WORK
sp
THROUGH
IT
THIS
sp
IS
sp
NOT
sp
THE
sp
CORRECTED
sp
VERSION
sp
YOU'VE
sp
GOT
sp
THE
sp
CORRECTED
sp
VERSION
sp
ERM
AND
THROUGH
sp
THIS
WE
CAN
SEE
WHERE
THE
sp
DIFFERENT
VARIATIONS
sp
COME
IN
THE
SYSTEM
sp
AND
INTRODUCE
ONE
OR
TWO
OTHER
ONES
sp
WHAT
WE'VE
GOT
sp
ALSO
IS
sp
INDICATED
A
BIT
OF
HOW
WE
MIGHT
sp
PROCESS
IT
IF
sp
WE
IMPLEMENT
THIS
SORT
OF
THING
sp
THE
sp
SORT
OF
IMPLEMENTATIONS
sp
WE'VE
sp
GOT
WE'VE
sp
GOT
AN
INPUT
PATTERN
sp
OKAY
WE'VE
GOT
sp
ER
sp
EXAMPLE
sp
P
sp
ER
SORRY
sp
CLASS
sp
EXAMPLE
sp
P
PLUS
R
sp
WE'VE
GOT
sp
A
T
sp
VECTOR
WHICH
IS
BASICALLY
AFTER
YOU'VE
TAKEN
THE
INPUT
sp
PATTERN
sp
JUST
AFTER
YOU'VE
TUPLED
sp
IT
sp
SO
WE
TAKE
OUR
INPUT
sp
PATTERN
AND
WE
PRODUCE
A
VECTOR
HERE
CALLED
T
sp
WHICH
WE
THEN
PRESENT
TO
THIS
sp
LOT
HERE
sp
WE'VE
GOT
A
WEIGHT
ARRAY
WHICH
sp
IS
THE
NORMAL
SORT
sp
OF
sp
THING
sp
WE'D
sp
HAVE
TRAINING
IS
A
SIMPLE
sp
P
SIMPLE
ALGORITHM
THIS
IS
HOW
sp
YOU
WOULD
ACTUALLY
IMPLEMENT
IT
sp
PROPERLY
I
MEAN
sp
IT
sp
INTRODUCES
A
AN
sp
INTERESTING
sp
THING
WHAT
WE'D
EFFECTIVELY
DO
sp
WHICH
{OOV}
CODE
sp
WHAT
EFFECTIVELY
DO
OF
COURSE
IS
WE
sp
WE
TRUNDLE
DOWN
HERE
sp
CHECKING
sp
TO
SEE
IF
ANY
OF
THESE
ARE
SET
TO
ONE
sp
AND
IF
THEY
ARE
SET
TO
ONE
sp
WE
SET
A
sp
WEIGHT
IN
MEMORY
sp
NOW
sp
IF
sp
YOU'RE
GON
NA
BE
WRITING
IN
SOFTWARE
YOU
DON'T
ACTUALLY
LOOK
AT
IT
sp
LIKE
THAT
sp
MAYBE
AND
IT
IS
ALSO
A
USEFUL
WAY
TO
sp
GENERALIZE
sp
THESE
DECODERS
sp
WHAT
THESE
DECODERS
ACTUALLY
DO
IS
TAKE
THE
sp
INPUT
sp
AND
ASSIGN
sp
ONE
sp
OF
FOUR
PARTICULAR
STATES
WHERE
sp
A
STATE
IS
sp
WHATEVER
THE
sp
PARTICULAR
SUB
sp
PATTERN
IS
ON
sp
HERE
sp
NOW
IF
YOU
VIEW
THE
OUTPUT
OF
THESE
AS
A
STATE
NUMBER
sp
YOU
CAN
GENERALIZE
sp
THE
sp
THE
BEHAVIOUR
OF
THESE
DECODERS
AND
YOU
CAN
GENERALIZE
sp
THEM
TO
ANYTHING
YOU
WANT
IT
COULD
sp
BE
AN
X
SQUARE
FUNCTION
THAT
PRODUCES
A
NUMBER
sp
WHICH
INDICATES
WHICH
sp
LINE
YOU
SHOULD
sp
ACTIVATE
sp
THAT'S
REALLY
HOW
THIS
IS
WRITTEN
DOWN
HERE
sp
YOU
GO
THROUGH
ALL
THE
EXAMPLE
PATTERNS
FOR
sp
EACH
THE
EXAMPLE
PATTERN
YOU
COMPUTE
sp
ER
THIS
sp
INPUT
sp
TUPLE
VECTOR
sp
SO
WE
TAKE
OUR
EXAMPLE
WE
PUT
IT
THROUGH
THESE
sp
DECODERS
sp
BEFORE
WE
DO
THIS
WE
HAVE
TO
APPLY
A
MAPPING
FUNCTION
sp
M
sp
THE
MAPPING
FUNCTION
IS
IMPORTANT
BECAUSE
AS
I
SAY
sp
THESE
WIRES
ARE
CONNECTED
sp
TO
THESE
PIXELS
HERE
sp
I
MUST
KNOW
EXACTLY
HOW
THOSE
ARE
CONNECTED
sp
IN
WHICH
sp
ORDER
BUT
sp
AS
WE'LL
SEE
LATER
THE
sp
DIFFERENT
SORT
OF
CONNECTIONS
THAT
YOU
HAVE
sp
ER
AFFECT
THE
BEHAVIOUR
OF
sp
IT
sp
ER
IF
YOU
GET
THE
CONNECTIONS
WRONG
sp
ER
SOMETIMES
IT
WON'T
RECOGNIZE
IT
sp
SO
THIS
MAPPING
FUNCTION
IS
QUITE
CRUCIAL
IN
IN
sp
THE
OPERATION
sp
SO
YOU
MAP
IT
sp
ONTO
THE
TUPLE
ARRAY
sp
AND
THEN
sp
ALL
THE
TUPLES
sp
SO
YOU
TAKE
EACH
TUPLE
sp
IN
sp
IN
ORDER
sp
YOU
SET
sp
THE
WEIGHT
ONE
sp
SORRY
THE
WEIGHT
IT'S
sp
PROBABLY
sp
IT'S
IT'S
sp
HARD
TO
EXPLAIN
IT
sp
LIKE
THAT
sp
BUT
sp
YOU
APPLY
sp
THE
sp
DECODER
FUNCTION
sp
F
sp
HERE
sp
F
sp
BRACKET
sp
TO
THE
PARTICULAR
TUPLE
sp
AND
YOU
GET
A
STATE
J
sp
SO
YOU
SET
THE
WEIGHT
sp
FOR
THAT
sp
PARTICULAR
STATE
sp
W
R
I
J
sp
THERE'S
sp
A
sp
J
THERE
sp
THERE'S
sp
A
sp
J
sp
THERE
OKAY
SO
EFFECTIVELY
WE
TAKE
OUR
INPUT
PATTERN
sp
WE
TAKE
OUR
TUPLE
WE
sp
FORCE
IS
sp
THROUGH
OUR
FUNCTION
sp
HERE
sp
WE
GET
OUR
VALUE
OUT
SUM
sp
VALUE
OF
sp
OF
sp
THE
TUPLE
sp
AND
sp
THAT'S
sp
J
sp
AND
sp
THAT
sp
INDICATES
sp
WITHIN
THE
WEIGHT
sp
ARRAY
THAT
WE'RE
LOOKING
AT
sp
WHICH
sp
WEIGHT
sp
WE
ACTUALLY
SET
sp
FUNNY
WAY
OF
LOOKING
AT
IT
I
KNOW
BUT
IT'S
SORT
OF
sp
TO
GIVE
YOU
AN
IDEA
sp
OF
HOW
sp
HOW
THE
DIFFERENT
WAYS
YOU
sp
CAN
sp
ACTUALLY
DO
THIS
sp
THIS
sp
MECHANISM
AND
sp
BRINGS
OUT
sp
THE
ER
VARIATIONS
sp
LOTS
OF
WAYS
OF
sp
WRITING
sp
IT
sp
TESTING
IS
THE
SAME
sp
SORT
OF
THING
sp
BUT
THIS
sp
BRINGS
OUT
THE
DIFFERENT
FUNCTIONS
sp
THAT
I
TALKED
sp
ABOUT
sp
EITHER
SUMMATIONS
OR
OR
sp
PRODUCTS
sp
DO
THE
SAME
sp
SORT
OF
THING
sp
YOU
sp
NEED
A
RESPONSE
ARRAY
FIRST
FOR
THE
RESULT
IN
sp
THIS
CASE
WE'VE
GOT
AN
UNKNOWN
sp
PATTERN
sp
WE
DO
EXACTLY
THE
SAME
sp
MAPPING
sp
OUR
INPUT
sp
PATTERN
sp
ONTO
sp
OUR
TUPLES
sp
WE
THEN
APPLY
OUR
sp
TUPLING
FUNCTION
F
TO
sp
GET
sp
J
AND
sp
THEN
sp
WE
sp
SUM
sp
FOR
ALL
TUPLES
sp
OVER
ALL
STATES
sp
THE
WEIGHTS
TIMES
sp
THE
sp
INPUTS
sp
A
sp
VERY
STRANGE
sp
WAY
OF
sp
WRITING
IT
AGAIN
sp
BUT
sp
BASICALLY
IT
sp
COMES
DOWN
TO
THE
sp
FOLLOWING
sp
AS
I
SAY
WE
TAKE
OUR
TUPLE
sp
PUSH
IT
THROUGH
OUR
DECODER
AND
GET
OUR
STATE
sp
HERE
sp
J
THAT'S
THE
EASY
BIT
sp
BUT
WHY
WRITE
IT
LIKE
sp
THIS
sp
COS
WHAT
WE'RE
DOING
IS
GOING
FOR
EACH
N
sp
TUPLE
sp
WE
GO
THROUGH
EACH
SET
OF
STATES
HERE
sp
MATHEMATICALLY
IT'S
VERY
DIFFICULT
TO
DO
THIS
YOU
COULD
USE
A
CHRONIC
ADULTER
TO
ACTUALLY
WRITE
IT
A
BIT
sp
EASIER
sp
BUT
sp
ER
I
DIDN'T
KNOW
ABOUT
THAT
WHEN
I
sp
WROTE
THESE
SLIDES
sp
MANY
MANY
sp
YEARS
AGO
AND
sp
IT'S
sp
CARRIED
ON
SINCE
sp
ER
WE
WRITE
IT
LIKE
THIS
sp
ERM
sp
SO
sp
YOU
ACTUALLY
SCAN
DOWN
HERE
AND
sp
WHERE
sp
YOU
sp
WHICH
IS
ABETTING
THE
STATE
ACT
sp
FROM
HERE
IS
sp
ONE
sp
YOU
sp
SET
THE
WEIGHT
IN
sp
THE
MEMORY
sp
A
HORRIBLE
WAY
OF
EXPLAINING
sp
IT
sp
BUT
sp
ER
sp
IT'S
SOME
sp
ATTEMPT
sp
AT
sp
SOME
SOME
sp
PRECISENESS
sp
OKAY
sp
AS
sp
I
SAY
THE
VARIATIONS
sp
ON
THE
SYSTEM
ARE
sp
ALTERATION
OF
THIS
SUMMATION
sp
IN
HERE
TO
A
sp
PRODUCT
sp
OR
sp
{OOV}
sp
AND
sp
HAVING
sp
THAT
sp
MATHEMATICAL
DESCRIPTION
ALLOWS
US
sp
TO
CHANGE
THAT
sp
FROM
sp
SUMMING
sp
ALL
THE
N
sp
TUPLE
sp
TO
DOING
A
sp
PRODUCT
sp
OF
sp
ALL
sp
N
sp
TUPLE
sp
WHAT
THAT
sp
BASICALLY
MEANS
AGAIN
sp
IS
THAT
sp
FOR
sp
THIS
PARTICULAR
BLANK
HERE
THAT
sp
WE'RE
LOOK
AT
sp
OKAY
THE
STATE
sp
VALUE
U
sp
MIGHT
BE
ONE
sp
THE
WEIGHT
MIGHT
BE
ONE
sp
OKAY
AND
WE
GET
THAT
OUT
OF
THAT
COS
sp
WE'RE
sp
JUST
GOING
THROUGH
ALL
sp
THESE
HERE
AND
SEEING
sp
WHICH
ONE'S
GOING
TO
BE
ONE
sp
WE
THEN
sp
INSTEAD
OF
ADDING
THOSE
sp
PARTICULAR
OUTPUTS
sp
WE
DON'T
sp
WE
WE
SUM
THEM
sp
DON'T
FORGET
THAT
EACH
OF
THESE
DECODERS
ONLY
PRODUCES
ONE
AC
sp
ACTIVE
OUTPUT
LINE
sp
OKAY
sp
SO
IN
THIS
CASE
THIS
sp
ONE
MIGHT
BE
ACTIVE
sp
CORRESPONDING
TO
THAT
sp
WEIGHT
THERE
THAT
sp
THAT
sp
MAKES
THAT
sp
WHOLE
BAN
BANK
sp
SET
sp
TO
sp
ONE
sp
THIS
sp
BANK
MIGHT
BE
SET
sp
TO
ONE
AS
WELL
OR
ZERO
AND
sp
THEN
YOU
SUM
THOSE
TOGETHER
sp
WHICH
sp
IS
THAT
SUMMATION
sp
THERE
sp
OR
YOU
CAN
DO
A
PRODUCT
THERE'S
A
sp
PRODUCT
sp
THERE
sp
BASICALLY
A
PRODUCT
IS
EQUIVALENT
sp
TO
AN
AND
A
sp
PRODUCT
sp
IS
A
sp
CONTINUOUS
sp
FORM
OF
AN
AND
sp
WE'LL
SEE
WHAT
WHAT
THE
BENEFIT
OF
sp
OF
sp
THAT
IS
IN
GENERALIZATIONSHIP
sp
IN
A
SECOND
sp
WHAT
I'VE
PUT
DOWN
HERE
sp
IS
THE
WAY
YOU
ALTER
THAT
sp
SIMPLE
DESCRIPTION
sp
TO
sp
GIVE
YOU
A
CONTINUOUS
sp
WEIGHT
sp
RATHER
THAN
A
BINARY
WEIGHT
sp
DURING
TRAINING
sp
WHAT
WE
DO
sp
IN
THE
TRAINING
ALGORITHM
IS
sp
IS
OF
sp
COURSE
SET
A
sp
A
LINK
HERE
OR
A
WEIGHT
sp
HERE
TO
ONE
sp
IF
sp
THE
OUTPUT
OF
sp
MY
DECODER
IS
SET
TO
ONE
sp
OKAY
THAT'S
sp
EASY
sp
IN
A
CONTINUOUS
FORM
INSTEAD
OF
JUST
SETTING
A
BINARY
LINK
HERE
WE
ACTUALLY
COUNT
sp
THE
sp
NUMBER
OF
TIMES
sp
THIS
PARTICULAR
LINE
sp
IS
SET
TO
ONE
sp
BASICALLY
WHAT
WE'RE
DOING
IS
TO
COUNT
sp
DURING
TRAINING
HOW
OFTEN
sp
A
SUB
PATTERN
sp
A
sp
TUPLE
sp
HAS
A
PARTICULAR
sp
STATE
sp
HAS
A
PARTICULAR
PATTERN
sp
AND
THAT'S
sp
VERY
USEFUL
IN
SORT
OF
MORE
STATISTICAL
sp
TYPE
sp
PROBLEMS
sp
ER
sp
AS
WE'LL
SEE
LATER
sp
SO
BASICALLY
WHAT
THAT'S
DONE
IS
INTRODUCE
sp
THE
VARIATIONS
sp
OF
THE
LETTERS
I'M
GON
sp
NA
GO
THROUGH
EACH
ONE
OF
THOSE
SORT
OF
VARIATIONS
IN
A
BIT
sp
MORE
sp
DEPTH
sp
ER
sp
LATER
ON
sp
BUT
sp
ER
sp
I
HOPE
TO
sp
GIVE
YOU
sp
AN
IDEA
sp
FOR
sp
THAT
sp
SO
LET'S
HAVE
A
LOOK
AT
IT
AND
SEE
HOW
IT
WORKS
I
MEAN
THE
BASIC
sp
MECHANISM
sp
IS
sp
VERY
sp
SIMPLE
AS
I
sp
SAY
FIRST
THING
WE
WANT
TO
sp
HAVE
A
LOOK
AT
sp
IS
sp
HOW
IT
GENERALIZES
AS
sp
I
SAY
THE
METHOD
sp
HAS
GOT
TO
RECOGNIZE
THINGS
IT
HASN'T
sp
SEEN
BEFORE
sp
LIKE
WITH
ALL
NEURAL
NETWORKS
NOW
NORMAL
NEURAL
NETWORKS
sp
PERFORM
GENERALIZATIONS
sp
BY
OKAY
sp
HAVING
SOME
SPACE
sp
YEP
sp
HAVING
OUR
EXAMPLES
HERE
AND
sp
OUR
EXAMPLES
HERE
AND
MAYBE
SETTING
A
LINE
sp
IN
sp
BETWEEN
sp
THAT
sp
SUCH
THAT
ANY
NEW
EXAMPLE
ONE
sp
DOWN
HERE
CAN
sp
BE
SEEN
AS
sp
BEING
BELONGING
sp
TO
THAT
SET
sp
OKAY
sp
NOW
sp
THINKING
ABOUT
WHAT
WE'VE
JUST
SEEN
sp
HOW
DOES
THAT
RELATE
TO
THAT
sp
VERY
DIFFICULT
sp
WE'VE
GOT
OUR
SINGLE
LAYER
NETWORK
sp
AS
WE
KNOW
sp
SO
WE
TAKE
ALL
OUR
INPUTS
sp
WE
sp
PUSH
THEM
THROUGH
THE
DECODERS
AND
sp
WE'VE
GOT
OUR
SINGLE
sp
LAYER
AFTER
sp
THAT
sp
BUT
THERE
IS
A
STRAIGHT
LINE
IN
THERE
SOMEWHERE
sp
BUT
sp
WE
DO
THIS
sp
PRE
PROCESSING
sp
NOW
THE
PRE
PROCESSING
sp
MAKES
THE
PATTERNS
sp
MAKES
sp
THE
sp
THING
LINEARLY
sp
SOLUBLE
sp
SO
IF
WE
ADD
AN
EXAMPLE
sp
USING
THE
N
TUPLE
sp
TECHNIQUE
sp
OF
sp
THE
EXCLUSIVE
OR
PROBLEM
sp
FOR
EXAMPLE
sp
ER
sp
LIKE
sp
THAT
sp
WE
WANT
TO
sp
SEE
HOW
THE
NETWORK
WOULD
SOLVE
THAT
sp
OKAY
sp
IN
TERMS
OF
A
SINGLE
LAYER
NETWORK
sp