all right good afternoon today's guest
speaker is Emmett Scheer Emmett is the
CEO of twitch which was acquired by
Amazon where he now works and Emmett is
going to do a new format of class today
and talk about how to do great user
interviews so this is the talking to
users part of starting to start up on
should be really useful thank you very
much for coming
contact server knows where I'm coming
from
from this we started or by sitter my
first startup with justin khan right out
of college we started this company
called Kiko calendar it didn't go so
well
uh I went alright we built it we we sold
it but we sold it on eBay so that's not
necessarily the end you want for your
startup and it was a it was a good time
we learned a lot learned a lot about
programming we didn't know anything
about calendars neither of us were users
of calendars nor did we during the
period of time we were thinking I talked
to anyone who actually did use a
calendar so that was a that was not
optimal
we got the build stuff part of the
startup down we did not get the talk to
users part the second startup we started
we used a very common trick that lets
you get away with not talking to users
which was that we were our own consumer
we we had this idea for a television
show justin.tv a reality show at Justin
Khan's life and we built a whole set of
Technology and website around the
reality show we wanted to run and so we
were the user for that for that product
and that's actually one way to cheat and
get away with not talking to many other
users is if you're just building
something that literally is just for you
you don't need to talk to anyone else
because you know what it is you want and
what you need but that's if you're
really limiting way to start a startup
most startups are not just built for the
person who is uh who is using them and
when you do that every now and then you
get really lucky and you are
representative of some huge class of
people who all want the exact same thing
you do but very often also that just
turns into a side project that doesn't
go anywhere so we kept working on
justin.tv for a while and we actually
achieved a good deal of success because
it turned out that there were people out
there who wanted to do the same thing we
did which was broadcast
I was live on the internet but the issue
with justin.tv the thing that the thing
that sort of kept us from achieving
greatness is we hadn't figured out yet
how to how to build towards anything
beyond that initial TV show we knew how
to build a great product actually if you
wanted to run a live 24/7 reality
television show about your life we had
the website for you we had exactly what
you needed but if we wanted to go do
more than that I'm going to open it up
to a broader spectrum of people a
broader spectrum of use cases we didn't
have we didn't have the insight to
figure that out because we weren't that
user and so at some point we decided to
pivot justin.tv we decide we needed to
go in a new direction
we thought we'd built a lot of valuable
technology but we hadn't identified the
use case that would let it get really
big and there are two directions that
seem promising one of them was mobile
and one of them was gaming and I led the
gaming initiative inside of the company
and what we did with gaming that was
very very different from what we'd ever
done before was we actually went and
talked to users because well I loved
watching gaming video I was very aware
than neither I nor anyone else in the
company knew anything about broadcasting
video games and so I was an throughout
the content I thought there was a market
there that was sort of the insight that
did I had that I think wasn't common at
the time which is how much fun it was to
watch video games quick show of hands
people know about watching video games
in the internet here okay I'm going to
assume that people listening to this
also know about it if you don't know
about watching video games the internet
you should go read about that because
it's sort of important context for the
stuff I'm going to talk about but the
main point is I thought that was awesome
but I did know anything about the side
of it that was really important which is
actually acquiring the content to start
broadcasting so we went out and we ran a
actually very large number of user
interviews we talked to a lot of people
and brought that data back and that
formed the core of all the
decision-making that was for the next
three years of product features on
Twitch was sort of the insights we got
from that and we continued to talk to
users and in fact built an entire part
of the company whose job it is basically
to talk to our users
which is a whole division that we just
didn't even have at justin.tv we had no
one at the company whose job it was to
talk to our most important users so so
that was twitch and I'm going to I want
to give you guys a little bit of a
little bit of an insight into with
twitch what what that what that meant
going to go talk to users so we
determined that the broadcaster's are
the most important people and the reason
we determined that was when we went and
looked into the market wheel and we
looked into what what determined why
people watched a certain stream renard
went to a certain website they would
just follow the content right you had a
you had a piece of content you loved and
the broadcaster would come with you and
that's actually the one really important
point about user interviews which is
that who you talk to is as important as
what questions you ask them what you
pull away from it because if you go and
talk to a set of users if we've gone and
talked to viewers only would have gotten
a completely different set of feedback
than talking to the broadcaster's and
talking to the broadcaster's gave us
insight into how to build something for
them that turned out to be strategically
correct I wish I could tell you the
recipe for figuring out who the target
user is for your product and who your
target user should be but there isn't a
recipe it comes down to think really
hard and use your user judgment to
figure out who you're really building
this for so what I want to do is a
little bit sling a little bit
interactive now which is we're going to
I got a bunch of ideas from from you
guys actually so sort of suggested ideas
and I'm going to pick one of them and I
want everyone to sort of sit down and do
do step one of this process for me right
now which is think about who would you
go ask about this like which people
where would you go to find the people
you needed to talk to about this
in order to in order to learn about what
you should build and so the idea we're
going to use is let me see here of these
ideas so here it's a lecture focus
note-taking app the idea is I don't
think that the state of the art for
note-taking is good enough yet and I
want to make a note-taking app that in
you know improves that experience makes
taking notes in class better or taking
notes while listening to a lecture
online
better so you know maybe it has
collaboration features maybe it like
helps you focus better somehow it has
multimedia enhancements I don't know
right all sorts of possible features but
that's the that's the idea so take like
take 120 seconds right now and think
about not what you would ask or what the
right features for this app is but who
would you talk to who should the who's
going to give you that feedback that's
going to tell you whether this is good
or not I actually mean it right now take
your laptop out like tight write some
stuff down think think about like the
you can it's good enough to like think
that in your head but actually like if
you actually just write it down and like
just come up with the five people you
talk to the five types of people you
talk to and who you think the most
important one was like actually doing
because there's nothing like actually
running through a practice of something
and trying to do it to actually get it
into your head the right way to do it
I'm gratified to hear clicking in the
keywords now if you're following along
at home pause actually do it think about
who would you talk to because that's a
that is the first question for almost
any startup that you need to answer is
like who is my user and and where am I
going to find them
all right that's like way shorter than
you normally use to think about this
problem it's actually a really tricky
problem and like figuring out where to
source the people is pretty hard but uh
we're going to move along anyways in the
in this highly abbreviated version of
learning how to build a product and run
a user interview so can I can I can I
get one volunteer from the audience to
come up and tell us what uh who you
would talk to and we'll talk about it
you guys are all pre-selected here you
go
I don't know how to turn this thing on
here we go so who do you talk to I would
definitely talk to college students
first obviously because we sit in a lot
of lectures and specifically I want to
talk to college students studying
different subjects to see if maybe you
know your English major if that makes a
difference versus you're studying math
or computer science in terms of how you
want to take notes in different lectures
and uh so you go talk to a bunch of
college students would you pick any
particular subset of college students
like sounds like you want to talk to all
college students or like a broader room
I want to talk to college students I'm
like I'm break down the divisions by
like people who study different areas
maybe and then also maybe what makes
sense for people who have like different
study techniques because some people
take a lot of notes some people don't
take that many notes but still jot stuff
down write something that that's really
good start like that's that is actually
obviously a group of users you want to
talk to especially if you're targeting
something at you know at college
students as the consumer and if you're
talking to college students as a
consumer the you're going to get a lot
out of students about what their current
note-taking habits are and you know what
they would be excited about one of the
problems with selling things to college
students is that college students don't
actually spend very much money it's
really hard to get you guys to open your
wallets especially if you want them to
pay for a school related thing I mean
people don't ever want to buy textbooks
right I think you probably probably all
use checker
that you know borrow from your friend or
whatever and so uh one of the like one
of the things with that you'd be missing
if you go after just the students right
is you want to figure out who who is the
most important person to this to this
habit if you actually add a note-taking
app my guess is for colleges the people
most likely to actually buy a
note-taking app that you guys would use
would be college IT right I mean
presumably for most for the most part
you're going to sell software to
students like the people who have to get
bought into that is usually the school
administrator so that would mean up that
we want approach if you like okay well
you presumably go talk to the college
students and you find out they don't
actually buy any note-taking software
right now at all I mean likely it's
possible they do in which case I'm I'm
totally wrong and this is why yes you
have to go talk to the users but you
then have to try to maybe try other
other air groups right so I would talk
to college I would talk to IT
administrators as well that's a that's
another area it's really promising you
might talk to parents right who spends
money on their kids education it's like
willing to pull their wallet out like
the you know parents of kids parents
forgets you're a freshman or going up to
college for the first time you need this
app to make your kid productive so that
they don't fail out of college and and
there's actually a lot of groups that
are potential that aren't necessarily
the obvious user but who are critical
critical to your app success potentially
and you when you yourse when you're at
the very beginning of a startup like
this when you're like you have this idea
that you think is awesome you want to
have that broadest group you possibly
can you don't just want to talk to one
type of person and learn that you want
to get familiar with the space when you
get familiar with the various kinds of
people who could be contributing all
right so let's let's have someone go up
and we're going to pretend we're going
to run this user interview so we're
going to talk to a college student and
try to find out what we should build you
know what we should get into this
note-taking app so so some another
volunteer place for running an interview
yes all right so uh hello hi I'm
Stephanie hi Stephanie nice to be um
welcome thank you for agreeing to do
this user interview with us so I I
wanted to
hear from you about you know what are
your note-taking habits like how do you
take notes today sure so I take notes
and variety ways I like to now because
speed and efficiency and just to come
back to it later it's easy for me to
just take notes on my laptop and so a
lot of those notes will be primarily
text-based
but in certain classes so for example if
I'm taking a history class most of it
will be in text but if I'm taking it
taking a physics class for example
they're gonna be more complex diagrams
different angles that I have to draw and
so that's a little harder it harder for
me to get what software do you use for
this stuff today I just do pen and paper
for that independent paper so you do a
combination you take notes with pen and
paper you take that with your computer
sometimes yeah and when you take notes
with when you take all these notes at
the end like do you actually review them
like you be honest do you actually go
back and ever actually look at these
notes the pen and paper not so much but
yes to the software base because it's
more easy to access and it's easier for
me to to share and collaborate and maybe
like even merge notes with classmates
and friends so what do you what do you
stick notes today on your computer um
Google Docs and Evernote Google Docs and
every hat and tell me more about like
why two things at the same time um so
every note is easy if I'm trying to just
collect it for myself I think um and yes
you can share but I think Google Docs
for me is easier to share and it depends
also if you know a friend has already
created a folder for example a guru ox
and I just have to add to that folder if
it's a group project for example first
as if it's for my personal use I tend to
go more to it so it sounds like you do a
lot of like note-taking collaboration
yeah
why was integrated okay what uh uh tell
me more about that like like do you take
us do you wind up taking most of the
notes most the value of the notes out of
notes other people take or is it mostly
your owner it's your view at the end of
the semester how does that work it's
mostly mine because I'm pretty picky
about the way I like things organized um
like design wise or formatting
even color I'm really particular with
and like the font that we use and that
really affects the way I study so um I
tend to like it to like to like to
personalize it
even after I merge so you you pull in
notes for other people but then you
merge them into into the mailing works
for me right awesome and if you uh if
you have Evernote notes and you have
Google Docs notes and you have pen and
paper notes once the semester is over
you ever go back to any of that stuff
where is it like or quarters are in
quarters here right once the quarter is
over uh you ever go back to any of that
stuff you ever um for classes not so
much um but if it's notes that I've
taken for like talks like these for
example or if it's like interview prep
that I'm doing I tend to go back because
it's things that I like to kind of keep
fresh in my mind and to help me prep fur
for future things that's interesting
tell me more about that like you take
notes not just in class yeah um so I
take notes to also just summarize main
points so if it's like inspirational
quotes for example from talks that I go
to like these and then like maybe I'm
going to an event we're actually going
to meet someone and it helps to actually
to think about and to remember and
recall what was shared at the time that
you know I attended the talk or
something awesome alright well normally
I'd actually dig into a lot more detail
there's a huge amount of like open
questions there so my mind after hearing
that stuff questions about which people
do you collaborate with questions about
whether or not you like what the volume
of notes are and like how how long of
note-taking stuff in just sort of
digging into like what the current
behavior is but like in the interest of
time and not like keeping everyone here
hearing about the intricacies of one
person's note-taking habits forever
we're going to move on but thank you
very much definitely appreciate that so
uh so that that's like that kind of
stuff you notice we're not talking about
the actual content of the app at all
like I'm not I'm not really interested
in features I don't really want to know
about what they the specific feature set
in Google Docs or Evernote
I might start digging in a little bit
more into which features actually get
used like if she's actively
collaborating you know is how does that
work I heard some interesting things
about Oh yet we use folders that's
interesting to me but the main thing
you're trying to do when you're running
these four
set of interviews is not necessarily get
like questions about like user flows and
like optimizing that or questions about
I like the specifics of of any of that
stuff
kind of can be distracting because users
think they know what they want but like
you you get the you get the horseless
carriage effect where you're you're
you're getting asked for a faster horse
instead of trying to design the actual
real solution to the problem if you
start asking people about features so
you want to stay as far away from
features as possible because the things
they tell you wind up almost feeling
overwhelmingly real when you have a real
user asking you for a feature it's very
hard to say no to them
because here's a real person who really
has this problem and they they're saying
build me this feature but as you start
to talk to lots of people and really get
a sense for with what their problems are
you figure out if this is actually
promising your air or not and like based
on what I heard there it's like starting
from that user interview I'm not
necessarily positive there is a problem
or there's at least there's a there's a
big enough problem that it's worth
building a whole new product for because
I didn't hear a lot of like things where
I'm where there was a a big blocker
where there something really wrong with
the way it the way it it was working and
unless I had some big idea I would take
that as a you know maybe a negative sign
but it doesn't necessarily mean that you
can't you can't move forward and keep
talking to more people because just
because you're the first person you
don't get anything out of it doesn't
mean there's not going to be a ton more
people who actually have a problem and
you once you've talked you about six
seven eight people
you're usually about done it's unlikely
you're going to discover a bunch of new
information there which is why it's
important to talk to different extremes
of people right go find people who are
at different different points because
practice is just six or seven Stanford
college students you're going to get a
very different response that need we
talk to six or seven high school
students or six to seven six seven
parents all right what's going to look
at the
so based on that though right I think
the I think it's possible you could come
up with a set of ideas right you have
this information about how someone takes
notes you've you've come up with
potentially when you came up with study
you had you had some ideas when you
heard this idea you had some ideas just
like how you could build something cool
and so if you're going to build just one
feature on top of Google Docs what would
that feature be right and that's for a
new product like this that might be a
good way to like get started thinking
about where to go which is I okay so
they're extensively using this thing
right now how can we make that
experience just one quantum better
something that would be really exciting
to this person to be one one step ahead
and so don't you take two minutes right
now and think about what that feature
might be actually like try to try to
come up with what uh what you might do
based on what you heard from from
Stephanie that could convince her to
switch away from her current
collaborative multi-person all working
together workflow on Google Docs to your
new your new thing that is has all the
features of Google Docs Plus this one
special thing that's like going to make
it that's going to make it more more
useful and convince them to stop using
the thing they're already using
awesome alright so I'm gonna invite our
third guest if you if you have something
up uh I don't want to put you on the
spot if you feel like you don't you're
not sure but uh yeah so what I is it on
yeah okay what I thought about was like
the the reason she uses ever know is
because like of like sticky note type
notes like like more thoughts and like
details so I feel like Google Docs has
like documents and not like smaller
notes so I feel like a feature that
would be like super like a mobile
version of Drive that doesn't like isn't
that clunky and like this will make you
make real documents could be like really
useful awesome alright that's a that's a
good insight right that's exactly what's
one of the things you get out of that
user interview and now you have this
idea right you've gotten this I guess
user fit feedback you got this idea what
if we had a Google Docs that had the
collaborative aspects and the group
aspects of that but where are you you
could pull in more little one-off notes
and it was it was designed more around
note-taking and so the question is now
once you have this idea which I think
it's that it's a vacuum perfectly
reasonable approach is this enough
is this something people would actually
switch just to have and the way to
validate there's two ways to validate
that one is if you're a cricket
programmer you can literally just go
build it and throw it out in the world
and see what happens and that's a that's
great and if that when that works that's
a that's an excellent way to approach it
but a lot of the time that one little
thing that's just a little bit better
might take you three months to actually
build something that's worthy of
actually using and so you actually want
to go out and validate that idea further
before you go ahead and start building
it and so you might take that idea you
might go back go back out and you know
you can sit down with uh with diagrams
you can you can draw what the what it
looks like draw the workflow and go
bring that in front of people but the
one thing you really don't want to do is
ask them this is this is just a sort of
a trap
want to warn you against doing is just
don't go out and say to come up with the
future idea and go out and ask people
are you I've got this I'd great idea for
a feature are you excited about it
because the the feedback you get from
users if you tell them about a feature
and ask them is this feature good is
often oh yeah that's great like that
sounds like such a good idea but when
you actually take that in front of
people and you actually build it you
then find out that well they thought it
was such a clever idea no one actually
like cares the switch to get it and so
the one question you can't ask is is
this feature actually good or not yes
yeah so Sam's asking you what's the
what's the minimum you can actually get
away with to validate given that you
can't actually just go and ask them is
this good or not and it's it's highly
dependent the answer that's highly
dependent on the particular feature but
usually the the best thing you can do is
is really just hack something together
right it's you find if your if your idea
is to build something on top of Google
Docs don't for your v1 go rebuild a
Google Documents but for notetaking
application I find a way to write a
browser extension that that that stuff's
just that little bit of incremental
feature in and and see if it's actually
useful for people go like action go find
a way to cheat is what it comes down to
because if you can't actually put it in
front of people it's really really hard
to to find that out for bigger things
we're actually trying to get people to
spend money it actually gets a lot
easier so if you're selling it it's
great actually sales is this cure-all
for this problem get people to put give
you their credit card and I guarantee
you they're actually interested in the
future it's one of the most valid
anything's you can do for a product is
go out there and actually get them to
commit to pay you up front and the
problem is when you're working on a
student note-taking app that's going to
be relatively hard because you probably
unless your ideas that you're actually
going to sell it it's probably something
where you're thinking at least the the
trial version is free and you're not
necessary
that much by trying to charge people
money but if you go out there and you if
you can get people to say hey I'm going
to I'm going to give you money the money
test is amazing it really does
clarify whether or not they're really
excited about it or not because if
you're not five dollars excited about it
you're probably not very excited about
it so the last thing I wanted to do is
actually work through with you guys what
happened at twitch so I brought some
slides of feedback um that I'd like to
get put up
that's my only slides for the for the
thing and we're it's it's what it is
it's it's it's representative excerpts
of twitch feedback I had a whole like 26
page document full of all the feedback
and then I realized that reading that
was going to be a little bit tedious and
there was no way I'd make it through it
in a lecture so pretend that like this
is stuff is all representative of like
lots of people said this kind of thing
out to to us when we ask them questions
and I've already pre condensed it for
you into the real feedback you got so
when we were working on twitch to go
launch it we wouldn't we talked to a
bunch of existing justin.tv broadcasters
and asked them about their experience
broadcasting what they liked about
broadcasting why they broadcasted what
they broadcasted what else was going on
in their life and the interesting thing
is when you talk to users of your
product your detail users of your
product they come back to you with
actually very detail things about
features because they actually get mired
in the feature and you have sort of read
between the lines but they ask to
profess things like I want to be way to
clear the banlist in my chat room like
this that was actually very common
request because there's a particular
issue with how our chat rooms worked
people would ask for the ability to edit
the titles of highlights after creating
them and and it's it's this was like
this stuff was really consistent as you
talk to broadcasters you probably talk
to 12 14 so I like that
broadcaster so the justin.tv gaming
platform we got we got all this feedback
and you know what else we have we have
your competitors have all these cool
features like polls and scrolling tags
you can personalize chat there and so we
have some positive feedback they're like
oh you guys don't have ads that's great
I did feel the banned trolls so much
yourself about chat a bunch of stuff
around interactivity with with a
interactivity with the with their
viewers and that was all really
interesting right so this is what this
is what the Justin TV broadcasters
wanted us to build
and this is what they with where they
felt pain using the using the product
and so if you thought that what we did
was go and address these problems you
would be wrong because actually people
who are using your service already and
I'm going to put up with all these
issues kind of kind of means that these
are probably not actually the biggest
problems because if you're going to
ignore the fact that you can't edit the
band lists and titles are uneditable and
there's no way to get trolls out of your
channel and you're using the service
anyways maybe those aren't huge problems
and so that sort of brings up another
really important point which is you have
to compare ah you think if the compare
groups of people and compare the level
which they they argue each other so if
you go to the next slide yes nice
we got competitor broadcaster feedback
which is really interesting so this is
stuff that you we heard a lot from
people who are using other broadcast
platforms they wanted to be able to
switch multiple people onto their
channel at the same time they complained
about us not having a rev share program
or they talked a lot about how they're
trying to make a living they really want
to make money pursuing this pursuing
this gaming broadcasting thing they
talked about a lot about video stability
our service wasn't good in Europe
specifically but but just globally video
stability was this huge huge issue for
them and if you compare and contrast
actually it was really different like
the things that people who didn't use
our service said about what they cared
about was completely different than the
things the people who are using the
service cared about and we focus on this
stuff because this was the software
it was so bad they weren't even willing
to use our service because of it and
most of them actually had thought about
it because we were our user base what
happened to be a very well-educated user
base in this in the area who knew about
all their options for for this and they
would they actually you know reaching
out to them meant that they'd probably
tried all four services and actually had
an opinion it's great when you can get
users who are that that informed then
that that they understand the space that
well and if you go to the I'm just going
to go to the next slide yeah here we go
um the other big thing we did that that
was really important was we talked to
non broadcasters um so we went out there
and we talked to all the people
who weren't using us or competitors and
in many ways those are the most
important people right talking to your
competitors that's a short-term win all
right someone's using a competing piece
of computing software unless you're your
piece of competing software or something
like Google which is a search engine
which everyone uses okay maybe then then
there are no non users to convert but in
the case of gaming broadcasting almost
everyone's a non user right the majority
of this is true for most new products
the majority of people you're competing
with are non users they're people who
have never used your service before and
what they say is actually the most
important what they say is is the thing
that blocks you from expanding from
expending the size of the market with
your features right if you all you do is
look at your competitors and yourself
and all you do is talk to your computer
people who use your competitors products
people use your products you can never
expand try not never but you're not
learning the things that help you expand
the size of the market you want to talk
to people who aren't even trying to use
one of these things yet who've thought
about it maybe but who aren't corn into
it so what did they say
my computer is it fast enough I'm
focused on training 12 hours a day for
the next tournament
I like making the perfect video and like
editing it and so I just upload things
to YouTube I don't do live streaming I
don't I I have no desire to go into that
space or this is actually particularly
in Korea this is a big problem once our
strategy gets broadcast in a major
tournament we have to start over we have
to like come up with an entirely new
strategy and so the last thing we ever
do want to do would be broadcast our
practice sessions are you crazy that's
going to hurt us in the next big
tournament and so this became this
became a big outreach program for us
trying to figure out how we can get
people over this we bought people
computers we we worked really closely
with gaming broadcast software companies
to help the who made the broadcasting
software to make that better we started
building broadcasting into games and
into platforms like we built
broadcasting into the Xbox we built
broadcasting the PlayStation 4 because
we want to need to overcome this issue
that like it was too hard
broadcasting wasn't uh wasn't possible
and so you sort of combine these
Frosty's are the three big groups we
looked at for broadcasting uh and you
combine that feedback and what it tells
you is not the features to build right
because the features they asked for
things like pull
things like a you know the ability to
have child account like child accounts
on your account we haven't built most of
that stuff but what was important where
were the the issues like the goals they
were trying to accomplish their people
wanted money
people wanted stability in quality
people wanted universal access for
viewers all around the world to be able
to watch them and so that became our
focus actually and we dumped almost all
of our resources into things that none
of know one ever mentioned in an
interview but those are the things that
actually address the problem and the way
you could tell that it worked is whereas
we we would build these things then we'd
go back to this exact same people we
interviewed and we'd say hey you told us
you really cared a lot about making
money well we've built you the
subscription program that will let you
make money and it it's it's astonishing
because most people aren't have never
had that experience actually they've
never talked to someone and said it
really great if your product edit
feature X and then and then like two
months later or a month later your
product actually has feature X or at the
very least a feature that addresses the
problem they brought up and so it was
actually the the people we converted
first to our product were the people
that we talked to about user research
they were the ones who were actually the
most impressed which is kind of fun but
it really worked because we picked
people who were representative and we
picked big broadcasters small ones
medium ones and we we made sure we were
addressing their concerns and that that
was completely different from how we'd
approach the problem on justin.tv vision
justin.tv when we try to do this we we
sat down we trolled through huge amounts
of data like we spent tons of time
looking at Google Analytics looking at
Mixpanel looking at in-house analytics
tools figuring out how people use the
service looking at where our traffic
came from completion rates on flows we
spent all this time doing that and that
that's good I mean you can learn things
from that I'm not telling you not to
look at your at your data but it doesn't
tell you where you need to go it doesn't
tell you what the problems are you need
to address and so we would just sort of
invent these ideas and justin.tv and
then nine times out of ten without
talking to someone that idea turns out
to be bad and that's one of the most
disappointing things about doing user
interviews and user feedback which is
why I think so many people don't
do it which is you're going to get
negative in news about your your
favorite pet feature most of the time
like you have this great idea and you're
going to talk to a user and it's going
to turn out that uh that nobody actually
wants that like no no one's actually
they're actually completely concerned
about completely different things and
they don't care about what you thought
was important at all and and that's a
little bit sad but just just think about
how sad you'd be in four months when you
launched that feature and it turns out
no one actually wants to use it so I
think that's about it for my the lecture
section of what we're talking about I
want take some questions from from the
audience
what do you see startups getting what's
wrong about like most heart stuffing at
all the ones do what are the most common
um that's a the most common mistakes are
showing people your product
don't don't show them your product it's
sort of like telling them about a future
you want to learn about what's already
in their heads you want to avoid putting
things there the other thing is uh
asking about your your pet feature
direction so if you think you want to
add add subscriptions to your product
going and asking people would you pay
for a subscription going and asking them
would you use this feature and I'd say
the the other big mistake people make is
talking to who's available rather than
talking to who they need to talk to
there's certain users that are really
easy to get at because they are say
members of your forum already right you
know what you have some product forum
and you go and you talk to the users on
that forum because they're they're easy
to get access to we spent like weeks
digging for identity information and
figuring out who these people were
so we could contact them so we could
talk to them because a lot of these
people weren't it wasn't obvious they
were just some user on a on a site that
site didn't support messaging there was
like no obvious way to interact with
them and so we spent a bunch of time
trying to network and find those users
and bring them on because if you if you
just talk to who's easy to talk to
you're not really getting uh getting the
best data the the fortunate side there
is that almost everyone is flattered to
be asked what they think and so most of
them will actually talk to you and tell
you things yeah
how hard was it to get buy-in
the rest of your company I mean like you
can go and be like whatever I'm in
charge so you're doing what I say but
it's probably not the best way to get so
how did you get invented that's a good
question so the the question is uh how
hard is it to get buy-in from the rest
of the company and how do you do it
um getting buy-in if you just go to them
and say I figured I talked to the users
I figured it out we have to build this
is really hard um because people don't
trust you there's something magic about
showing them the interview though so I
really recommend you record interviews
recording interviews is like magic eh it
stops you from taking notes in the
middle and taking notes is a little bit
disruptive it makes it hard for you to
feel like you're actually engaged in the
conversation and be you can then play
that recording for people so in they
don't have to be there for the entirety
of all the interviews but when you want
to make a point about what what what we
should be building and why you can just
play back for the rest of the company of
that interview and it's like magic the
influence it has on people's thoughts
and what's what the right thing the
build is yes since you mentioned
recording did you try to something again
over an hour
yes so you definitely want to do Skype
up or start the question was to be
insist on Skype interviews for recording
you don't want to do interviews over
email if you can avoid it because
interviews over email are not
interactive and the most interesting
things you learn in interview is come
from the interesting tell me more
because the incident that you hit you'll
hope this vain you know they'll say
something you didn't expect and the
instant they say something you didn't
expect or didn't already know you should
drop into the Detective Mode and
Detective Mode is huh that's interesting
can you tell me more about that how many
people don't like silence so they'll
keep talking to fill the void and the
best part about doing it over Skype or
doing it in person is you have that
interactive feedback and you can
actually pull a lot more out of people
email interviews are they're okay but
they're they're basically useless if
you're in person or over Skype they're
actually also easy to record make sure
you ask them if it's okay to record it
it's not polite to record people without
their consent but if they're willing to
like give you an user interview they're
probably willing for you to record it as
well
like you mentioned it that you've read a
lot of beauties in Korea yeah so the
question is like what about people in
the international market we're trying to
do user interviews with you who don't
speak your language that's just really
hard and actually to this day twitch
works way better in english-speaking
countries than it does in non-english
speaking countries and I think a big
part of that is we are much better at
talking to people in english-speaking
countries and learning what their needs
are and we're not as good at it in other
countries we've tried to address that by
hiring people who speak Korean having
them translate we tried to address that
by finding representative people in
those countries who speak both English
in Korean and reaching out to them but
the problem with that is like the you're
not actually getting a representative
sample no matter how hard you try the
very fact that they are fluent English
speaker means they're not representative
of all the people who don't speak fluent
English it's just a hard problem it's
why companies find it easier to build
markets that win in their home in their
home country
much more easily than abroad because
it's really hard to talk to users abroad
yes
so the channels we use to what channels
to be used to reach out to them when do
we ever compensate them the channels we
use to reach out to them were on site
messaging systems so like if you're most
site websites have some way to contact
the user so they're visible user of
another web site you use that sites
messaging system and say hey I was
watching your stream or whatever this
person was doing on the site I'd love to
ask you some questions about your use
would you mind hopping on a Skype call
and as for the other thing we do is we
find out who peoples were me to send
them email we'd like running them at
events because a lot of these people go
to the same events and we like would go
to the events and like get we wouldn't
run the user interview at the event but
you get to know them you exchange
business cards or you know whatever it
is you actually do now that aren't isn't
business cards and uh and you get in
touch with them we tended not to
compensate people I think that if you if
note if people don't care enough about
the problem to like talk to someone
who's trying to solve it you're probably
barking up the wrong tree we never had
any trouble getting people to talk to us
without paying
so so this is whole second set of user
feedback that's really important that
actually talked about the question was
what about like on-site user feedback
tools and I think the stuff you're
talking about is where you have like a
new product and you want to see how if
it's actually going to work or not and
so you put it in front of people and you
see how they use it or not that's really
important that kind of work is super
super important it can tell you lots of
things about where you went wrong
building something before you launch it
which is great it doesn't tell you what
to build it helps you iron out the kinks
and edges of the thing you did build but
generally speaking we uh that wasn't the
kind of user feedback we were getting I
mean if that stuff's good it's good it's
like it's much more similar though to
the the data driven approach right
you're finding out why are people
dropping off in this flow you're not
finding out what problem should I really
be solving for them and what what do
they care about as a human and for this
kind of like really early says user
interview which is the kind of user
interview that's crucial startups do
that's the that's where you want to
focus so we didn't bring anyone on site
action it was almost all over phone or
Skype yes so treated groups of people
there is no kind of feedback so at the
time yeah so with the three different
kinds of people to be focused on one of
them given that we had very limited
resources yes we focused on the
competing people using competing
products because we knew that they
already were interested in the behavior
that we needed and uh they were willing
to do it at all and therefore all we had
to do is convince them to switch which
is a much easier thing to do than to try
to create a new behavior where none
existed before and we had to do that
because we had to get some quick wins
because my gaming project inside of
justin.tv would have been killed if it
wasn't showing 25 percent
month-over-month growth every single
month so we did and that meant focusing
on short-term get the people in right
now let's work out to be good in general
because it turns out that building
something that some people want
generally generalizes and so I want to
bring in people who weren't even users
of the service as well
yes
she's gonna rock in the beginning so I
build up for example the video game
industry then beginning this industry
was very decentralized like there wasn't
a lot of cohesion like different video
game companies consolidating where
tournaments are and stuff but now it's
very different so originally spoke to a
fraud casters and you know strangers
themselves how is that changed when like
for example like riot has you know
banned users from our professional
pleasures from streaming their own stuff
try to you know gain leverage with that
yeah so the question is what about the
game publisher it's basically right the
game publisher is these huge important
people in the space hey the game
publishers and any big company for that
matter isn't going to give you the time
of day as a small startup which is both
good and bad it means you don't really
need need to talk to them because
they're uh they're not interested in you
but it means you actually just can't
talk to them I mean we tried but no one
wanted to talk to us and they did once
we started getting some traction and and
becoming a little bit slightly bit of a
player in the space I don't like talk
that bad about them because I they they
were nice about enough about it it's
just that you know when you're when
you're a tiny little startup there's
lots of tiny little startups and they
don't have the time to talk to all of
you as we've gotten bigger actually the
the point that you know game publishers
have become an increasingly important
constituency for us and by ways to talk
about who twitch does user interviews
with now whoo-wee whoo-wee pulled
information from now it would include
game publishers definitely these they've
become much more active in the space it
was something that they weren't
particularly active three or four years
ago as much as they are now and that's
another real important point about user
interviews in general which is that the
pool people you care about is going to
shift over time the people who get you
started like the crucial people to get
your product started for the first six
months are not who will be using it
three years later and it's very
important you keep doing this stuff
because one thing that's really easy to
do is do a little bit of it in the
beginning and and achieve some level of
success and then you're sort of stopped
talking to new people and that's a good
way to make the next set of features you
build be not as good as the first ones
yeah yes
how do you give good user feedback if
you're a user um so how do you give good
user feedback is really a question so uh
I think what
what I want a user to do is I want to
user to tell me about uh what they like
what they're really thinking right and
what what what their problems really are
and to just sort of ramble like I want
someone to like just tell me about stuff
in their life because the the more you
learn about them as a person and sort of
that the their what's going on in the
context of what they're doing the easier
to understand why they want the things
they want and that's really the critical
question so I'd say like you know what
I'm looking for an A in someone when I'm
doing a user interview it's someone
who's going to be willing to talk a lot
and be willing to really give me a full
give me a full picture so that's what I
guess on the flip side is going to be a
good you want to help people out with
good user interview feedback ramble like
people just just talk about stuff and
everything alright great well thank you
very much