Incident Post Mortem: November 19, 2021


Summary


On
November
19,
2021,
Coinbase
learned
that
it
had
erroneously
credited
some
customers
transacting
in
GYEN
and
POWR
either
100x
or
1/100th
the
amount
they
purchased.
Coinbase
promptly
disabled
trading
in
POWR
and
GYEN,
worked
around
the
clock
to
resolve
the
underlying
technical
issue,
and
then
made
adjustments
in
customer
accounts
to
reflect
the
amount
of
GYEN
and
POWR
that
customers
actually
purchased.
This
incident
affected
approximately
0.0072%
of
Coinbase’s
total
verified
users.


What
happened?

On
November
19
at
approximately
4:00
p.m.
EST,
Coinbase
updated
an
internal
data
source
related
to
POWR
and
GYEN
precision.
The
update
was
tested
through
our
standard
automated
testing
and
deployment
monitoring
procedures.
However,
the
testing
didn’t
detect
that
the
update
would
propagate
at
various
speeds
through
a
number
of
internal
systems
and
would
result
in
customers
being
credited
either
100x
or
1/100th
the
amount
of
GYEN
or
POWR
they
purchased.

The
data
rollout
error
was
identified
through
our
position
risk
monitoring
systems
shortly
after
the
November
19
4:00
p.m.
EST
update.
At
5:35
p.m.
EST,
we
disabled
transacting
in
GYEN
and
POWR
pending
resolution
of
the
underlying
issue.
At
7:26
p.m.
EST,
we
identified
accounts
that
transacted
in
GYEN
or
POWR
during
the
data
rollout,
and
temporarily
restricted
these
accounts
pending
further
investigation.
By
November
21,
restrictions
were
removed
for
98.8%
of
these
accounts
and,
by
December
13,
Coinbase
restored
full
trading
for
GYEN
and
POWR.


What
did
Coinbase
do
to
correct
the
problem?

Coinbase
immediately
devoted
substantial
engineering
resources
to
quickly
correct
the
problem,
ensuring
our
customers
received
the
correct
amount
of
GYEN
and
POWR
that
they
purchased.
For
customers
who
were
erroneously
over-credited
100x
the
GYEN
and
POWR
they
purchased,
we
ensured
that
they
received
the
correct
amount
of
assets
that
they
paid
for.
For
those
who
still
had
GYEN
and
POWR
in
their
accounts,
this
was
relatively
straightforward

we
notified
customers
of
the
error
and
simply
debited
those
customers’
accounts,
removing
the
extra
GYEN
or
POWR
that
was
erroneously
credited.

Some
customers
had
already
converted
their
GYEN
and
POWR
to
other
digital
assets,
such
as
Bitcoin.
Other
customers
sent
their
GYEN
and
POWR
to
wallets
off
the
Coinbase
platform,
but
kept
other
digital
assets
on
the
Coinbase
platform.
For
these
customers,
we
notified
them
of
the
error
and,
in
accordance
with
the
Coinbase
User
Agreement,
withdrew
other
assets
from
these
customers’
Coinbase
accounts
equal
to
the
amount
of
GYEN
or
POWR
they
had
been
over-credited.

When
determining
how
much
to
debit
from
these
customers’
accounts,
we
used
the
most
favorable
exchange
rate
for
our
customers.
Specifically,
we
calculated
the
USD
value
of
the
GYEN
or
POWR
owed
to
Coinbase
by
using
the

lowest

exchange
rate
on
the
Coinbase
Exchange
from
the
time
this
incident
began
until
trading
was
halted
($0.00825/GYEN,
$0.4742/POWR).
This
minimized
the
amount
owed
to
Coinbase
by
these
customers.
We
then
debited
funds
from
user
accounts
up
to
this
USD
value,
starting
with
their
fiat
balances,
then
USDC
and
other
stablecoin
balances,
followed
by
other
digital
asset
balances
ranked
by
descending
market
cap.
The
value
of
these
digital
assets
was
calculated
using
the
market
rate
at
the
time
user
accounts
were
debited.

A
small
group
of
customers
who
were
erroneously
over-credited
GYEN
or
POWR
sent
these
digital
assets
off-platform
and
left
no
other
assets
on
the
Coinbase
platform.
Coinbase
has
been
reaching
out
to
those
customers
individually
and
appreciates
our
customers’
cooperation
returning
the
erroneously
credited
GYEN
and
POWR.
Repayment
of
the
over-credited
funds
is
required
under
the
Coinbase
User
Agreement.

For
customers
who
were
undercredited
GYEN
or
POWR,
receiving
a
lower
amount
than
they
purchased,
Coinbase
first
determined
the
amount
of
GYEN
or
POWR
owed
to
these
customers.
Coinbase
then
calculated
the
USD
value
of
the
GYEN
or
POWR
owed
to
customers
by
using
the

highest

exchange
rate
from
the
start
of
the
incident
until
the
remediation
process
was
completed
($0.009799/GYEN,
$0.9617/POWR),
which
was
the
most
favorable
exchange
rate
for
our
customers.
That
means
that
regardless
of
the
price
customers
purchased
at,
we
assumed
that
the
customers
would
have
sold
these
assets
at
the
highest
price
while
trading
was
disabled.
After
calculating
this
USD
value,
we
credited
customers
an
equivalent
amount
of
Bitcoin.
We
credited
these
customers
in
Bitcoin
because
GYEN
and
POWR
trading
was
still
suspended,
and
Bitcoin
is
used
in
every
country
where
customers
were
affected.

To
further
benefit
our
customers
we
used
an
exchange
rate
of
$55,000/BTC,
which
was

lower

than
the
market
rate
of
BTC
at
the
time
these
BTC
payments
were
made.
This
exchange
rate
ensured
our
customers
received
more
Bitcoin
than
they
would
have
received
had
we
used
the
actual
BTC-USD
exchange
rate
at
the
time.

Additionally,
for
all
customers
whose
accounts
were
restricted,
Coinbase
provided
a
customer
experience
credit
of
up
to
$100
in
BTC.


What
happens
next?

Many
customers
still
have
questions
about
how
their
accounts
were
credited
or
debited
because
of
this
incident.
For
questions
specific
to
your
account,
please
feel
free
to
reach
out
to

Coinbase
Support
.

Coinbase
is
also
revising
the
information
in
our
customers’
account
statements
and
tax
forms
to
correctly
reflect
our
customers’
GYEN
and
POWR
transactions.
If
your
statements
or
tax
forms
appear
incorrect,
please
reach
out
to
Coinbase
Support,
but
know
that
we
are
working
to
correct
that
information
as
well.


GYEN
Values
Before
the
Data
Rollout
Error.

In
the
days
leading
up
to
the
data
rollout
error,
between
November
16
and
November
19,
Coinbase
Exchange
observed
GYEN-USD
break
parity
when
compared
to
JPY/USD.
We
have
seen
customers
speculate
on
social
media
that
this
incident
was
somehow
related
to
this
break
in
parity.
We
have
also
seen
customers
speculate
that
this
break
in
parity
was
somehow
caused
by
Coinbase.
These
allegations
are
false
and
reflect
a
misunderstanding
about
what
GYEN
is
and
how
Coinbase
works.



The
Break
in
Parity
Occurred
Before
and
Was
Not
Related
to
the
Incident.

This
break
in
parity
occurred
days
before
the
incident.
At
the
peak
of
this
break
in
parity,
on
November
17,
1
GYEN
traded
for
approximately
¥7.48.


The
price
of
GYEN
(blue)
in
Yen
(red)
during
the
break
in
parity,
with
a
high
of
¥7.48
(Red
Line
Indicating
the
time
of
the
Data
Rollout
Error)

By
the
time
the
data
rollout
error
occurred,
on
November
19,
GYEN’s
price
stability
had
recovered
and
GYEN
was
trading
at
approximately
¥0.96–0.98.
When
Coinbase
unrestricted
impacted
customers’
accounts,
GYEN
was
trading
at
approximately
¥0.98.
In
other
words,
the
break
in
parity
occurred
before
the
data
rollout
error
and
the
two
issues
had
nothing
to
do
with
each
other.



How
the
Break
in
Parity
Occurred.

When
Coinbase
listed
GYEN,
there
was
significant
demand
for
GYEN
that
could
not
be
matched
by
supply.
The
surge
in
buyer
demand
for
GYEN,
coupled
with
the
insufficient
supply
of
GYEN
across
all
markets
(not
just
Coinbase),
ultimately
caused
the
break
in
parity.
From
November
17
through
November
19,
Coinbase
implemented
an
alert,
informing
its
customers
who
were
buying,
selling
and
trading
GYEN
of
“Unusual
Market
Activity

Due
to
unusual
market
activity
for
GYEN,
you
may
have
trouble
trading
GYEN
on
Coinbase.com.
We
apologize
for
any
inconvenience
caused
by
this.”
The
break
in
parity
occurred
because
of
these
market
conditions
specific
to
the
GYEN
digital
asset
unrelated
to
Coinbase
operations.

Check out the original article here.
Author: Coinbase