Building entropically, Gemini, Prompt engineerings revenge, and Superpowered

1
00:00:00,000 --> 00:00:05,029
Nabeel Hyatt: technology tends to go
through ages of entropy and de entropy.

2
00:00:05,569 --> 00:00:11,229
We all love, especially as engineers,
we love de entropy, we love simplifying

3
00:00:11,229 --> 00:00:15,778
everything, cleaning it up getting
the signal from noise bringing it

4
00:00:15,778 --> 00:00:17,778
all down into something that works.

5
00:00:17,998 --> 00:00:22,328
Things that are trying to make a promise
of de entropy too quickly when all

6
00:00:22,328 --> 00:00:26,414
of these LLMs are so new Just feel
incongruous to me when the goal is

7
00:00:26,414 --> 00:00:30,751
solve the problem reliably and, and
we're still not at reliable solution.

8
00:00:31,151 --> 00:00:33,891
Fraser Kelton: Boy, we wrestled with this
one, but that one feels really right.

9
00:00:34,426 --> 00:00:38,426
It's going to get more complicated
in every direction because we are

10
00:00:38,426 --> 00:00:44,186
not at the reliability required for
consistent value in many use cases.

11
00:00:44,676 --> 00:00:49,626
And like, why bother adding abstractions
of simplicity if you say it's

12
00:00:49,626 --> 00:00:50,646
still not going to be good enough?

13
00:00:51,416 --> 00:00:52,186
Nabeel Hyatt: Yeah, exactly.

14
00:00:52,196 --> 00:00:55,106
Fraser Kelton: lot easier for you to get
something than broken into production.

15
00:00:56,136 --> 00:00:57,746
Nabeel Hyatt: That's
what, that's the headline.

16
00:00:58,036 --> 00:01:00,616
Why are we making it easier to
get broken things into production?

17
00:01:00,906 --> 00:01:02,186
Fraser Kelton: Oh, but what a teaser.

18
00:01:02,531 --> 00:01:03,501
Nabeel Hyatt: I know, I know.

19
00:01:04,221 --> 00:01:05,041
Hello everybody.

20
00:01:05,271 --> 00:01:06,651
Welcome to Hallway Chat.

21
00:01:06,711 --> 00:01:07,621
I'm Nabeel.

22
00:01:08,559 --> 00:01:09,139
Fraser Kelton: Fraser.

23
00:01:09,644 --> 00:01:12,644
Nabeel Hyatt: And we are here to
talk about what we've been talking

24
00:01:12,644 --> 00:01:14,794
about in the world of AI mostly.

25
00:01:15,174 --> 00:01:17,794
I, I didn't really know we were signing
up for this every week when we signed up.

26
00:01:18,284 --> 00:01:19,604
They come fast, Fraser.

27
00:01:20,604 --> 00:01:23,074
I felt like I just talked to you
on the hallway chat last week

28
00:01:23,094 --> 00:01:25,954
about the launch of ChatGPT and
all of your stories around that.

29
00:01:26,319 --> 00:01:29,479
But at the same time, there's like
a million things to also talk about.

30
00:01:29,479 --> 00:01:32,249
So it is both, feels like these
C shows are coming all the time,

31
00:01:32,249 --> 00:01:34,229
but also too much to talk about.

32
00:01:35,369 --> 00:01:38,459
Fraser Kelton: I was thinking after
recording the last one, how nice it is to

33
00:01:38,469 --> 00:01:43,669
be able to talk in depth with you about
these topics and just laugh and explore.

34
00:01:43,699 --> 00:01:45,099
And so I'm good with it all.

35
00:01:45,379 --> 00:01:47,799
The, you know, here, here's
something last week, I think I

36
00:01:47,829 --> 00:01:49,279
said the line, where's Google.

37
00:01:49,984 --> 00:01:51,164
And we had an answer.

38
00:01:51,834 --> 00:01:52,694
Kind of, right?

39
00:01:52,694 --> 00:01:53,944
We kind of had an answer.

40
00:01:54,279 --> 00:01:54,829
Nabeel Hyatt: Yeah.

41
00:01:55,189 --> 00:01:59,879
Oh, I loved, we had our AI dinner this
week, and we had somebody from the

42
00:01:59,889 --> 00:02:04,556
Gemini team sitting at the dinner,
all night long, mouth shut, and I

43
00:02:04,556 --> 00:02:08,089
am like spouting off and spitting
all kinds of stuff about Google.

44
00:02:08,409 --> 00:02:11,139
And I don't know that I picked
up the smug look on his face.

45
00:02:13,034 --> 00:02:13,424
Fraser Kelton: I

46
00:02:13,492 --> 00:02:16,492
Nabeel Hyatt: yep, tomorrow morning
you're gonna see that Google's

47
00:02:16,522 --> 00:02:17,752
got a little bit of a comeback.

48
00:02:17,852 --> 00:02:19,942
Although I think it's a little
bit of a comeback, right?

49
00:02:20,012 --> 00:02:20,352
So...

50
00:02:20,352 --> 00:02:21,648
what is Gemini, Fraser?

51
00:02:22,403 --> 00:02:28,663
Fraser Kelton: Gemini gets announced
in summer by Google, where they say,

52
00:02:29,283 --> 00:02:35,763
we're training, A large language
model that's going to be amazing.

53
00:02:36,343 --> 00:02:41,133
And has now become a little bit of a meme
because they have talked and talked about

54
00:02:41,133 --> 00:02:44,033
what they're about to do and the rumors.

55
00:02:44,243 --> 00:02:46,113
Nabeel Hyatt: their answer to
OpenAI and Anthropic and the others.

56
00:02:46,213 --> 00:02:46,523
Yep.

57
00:02:47,023 --> 00:02:47,893
Fraser Kelton: Yep, that's right.

58
00:02:48,143 --> 00:02:51,643
And we're at that dinner, as you
say, and it's become a little bit

59
00:02:51,643 --> 00:02:53,123
of a joke as like, where are they?

60
00:02:53,343 --> 00:02:55,383
They've been talking about this
for months and nothing's here.

61
00:02:55,403 --> 00:02:57,693
And then, boom, we wake
up the next morning.

62
00:02:58,083 --> 00:03:02,443
I get a text from my friend
that says, surprise, and

63
00:03:02,443 --> 00:03:04,953
they've shipped some of Gemini.

64
00:03:05,113 --> 00:03:09,913
They haven't shipped the most capable
model and they shipped a lot of.

65
00:03:10,343 --> 00:03:15,053
Demo videos which we'll come back to
and talk about a little bit, but they've

66
00:03:15,143 --> 00:03:19,133
announced a something called Gemini Ultra,
which is, you can think of it as the

67
00:03:19,133 --> 00:03:24,093
equivalent of GPT 4,, then they've shipped
Gemini, I don't know, the terminology is

68
00:03:24,093 --> 00:03:29,403
crazy, Pro, Gemini Pro, and Gemini Ultra
is not available, Gemini Pro is available

69
00:03:29,433 --> 00:03:34,853
as of the day of the launch, and that
is comparable in performance, at least

70
00:03:34,853 --> 00:03:38,153
on the evals, the evaluations, to GPT 3.

71
00:03:38,903 --> 00:03:43,288
5, and then they have What I think
is probably maybe one of the more

72
00:03:43,288 --> 00:03:46,828
interesting things, they've then
distilled it all down to something

73
00:03:46,828 --> 00:03:53,351
called Nano, which can run on, and is
running on the Pixel, which is a pretty

74
00:03:53,351 --> 00:03:55,501
awesome thing for them to have done.

75
00:03:55,951 --> 00:04:02,021
Worth calling out, Gemini Pro, the
mid tier model that's equivalent to 3.

76
00:04:02,061 --> 00:04:09,151
5, is now live and integrated into
BARD, their ChatGPT like product.

77
00:04:09,631 --> 00:04:13,631
And it's, you know, one year after
the fact that that's been rolled out

78
00:04:13,641 --> 00:04:17,421
broadly by OpenAI, and so quite a

79
00:04:17,536 --> 00:04:20,796
Nabeel Hyatt: I have to admit,
Fraser, If this is nothing else,

80
00:04:20,866 --> 00:04:22,986
it reminded me that BARD exists.

81
00:04:23,666 --> 00:04:26,186
It was the first time
we've even mentioned BARD.

82
00:04:26,626 --> 00:04:30,606
Like that, that, how useful it is
in at least our workflows, right?

83
00:04:31,876 --> 00:04:33,176
Fraser Kelton: it has
reminded me that it's a thing.

84
00:04:33,176 --> 00:04:34,766
It still didn't encourage me to go use it.

85
00:04:34,766 --> 00:04:38,506
Like, I'm not sure what this is
an answer to in terms of ChatGPT.

86
00:04:38,536 --> 00:04:40,436
So, they now have a parity with the model?

87
00:04:40,496 --> 00:04:41,016
Okay.

88
00:04:41,166 --> 00:04:41,606
Right.

89
00:04:41,856 --> 00:04:42,346
Okay.

90
00:04:42,666 --> 00:04:43,046
Sure.

91
00:04:43,371 --> 00:04:45,691
Nabeel Hyatt: We were texting and
I've got some WhatsApp groups that

92
00:04:45,691 --> 00:04:49,321
were fiddling around and talking about
this when it launched in a Discord

93
00:04:49,321 --> 00:04:51,481
group of AI engineers and so on.

94
00:04:51,801 --> 00:04:56,611
And I got to say the evals of course
went from oh my god and then the

95
00:04:56,611 --> 00:05:02,591
demo videos, oh my god, to pretty
quickly, hey, is this a bunch of BS?

96
00:05:02,931 --> 00:05:03,961
Fraser Kelton: Yeah, that's right.

97
00:05:04,526 --> 00:05:08,936
I think the first thing to call out
is I think the general level of

98
00:05:08,936 --> 00:05:14,481
performance across Some eval benchmarks
give you a sense of where relative

99
00:05:14,541 --> 00:05:16,781
performance of the model might be, right?

100
00:05:16,781 --> 00:05:22,611
So GPT 4, far outperforms basically
anything up until the release of

101
00:05:22,621 --> 00:05:26,481
Ultra, and you could then have probably
high confidence that it's going to

102
00:05:26,481 --> 00:05:29,691
perform in a very different class
once you get it into production.

103
00:05:29,691 --> 00:05:32,811
I think the thing that we can say is
without having actually played with it,

104
00:05:32,821 --> 00:05:39,501
the evals suggest that Ultra, the large
one is directionally equivalent to GPT 4.

105
00:05:39,501 --> 00:05:41,741
And That, that's something, right?

106
00:05:41,741 --> 00:05:45,911
I think that the fact that we might
now have two GPT 4 type equivalent

107
00:05:45,911 --> 00:05:50,531
models in, in market proves that
somebody else other than OpenAI can

108
00:05:50,531 --> 00:05:52,191
do something on this magnitude and,

109
00:05:52,336 --> 00:05:53,536
Nabeel Hyatt: not willing say that yet.

110
00:05:53,966 --> 00:05:56,276
That's I'm not absolutely crap.

111
00:05:56,496 --> 00:06:00,606
Like I'm willing to say that
soon and, and, but, but,

112
00:06:00,941 --> 00:06:01,731
Fraser Kelton: Yeah, that's fair.

113
00:06:01,823 --> 00:06:07,403
, Nabeel Hyatt: the MMLU benchmarks are
comparing a five shot reported GPT 4

114
00:06:07,743 --> 00:06:11,643
benchmark to a 32 shot, I think it was.

115
00:06:12,103 --> 00:06:14,153
If I remember correctly, UltraReport.

116
00:06:14,483 --> 00:06:17,973
It's just not, those are
not comparable at all.

117
00:06:18,123 --> 00:06:22,053
Frankly, the stuff that everybody would
probably normally out of the box use

118
00:06:22,053 --> 00:06:26,203
this product for, the average consumer,
those all, at least the evals we can

119
00:06:26,203 --> 00:06:28,793
see, seem comparable and seem fine.

120
00:06:28,823 --> 00:06:33,863
So let's not overstate some kind
of eval problem where there isn't

121
00:06:33,863 --> 00:06:35,503
one, least in this specific case.

122
00:06:35,883 --> 00:06:39,383
But, the kind of like general
math cases in particular seemed

123
00:06:39,543 --> 00:06:41,423
a little cooked and unfortunate.

124
00:06:41,453 --> 00:06:45,183
And I think, frankly, when they're
speaking to a highly technical audience,

125
00:06:45,583 --> 00:06:49,193
I'm not sure why they were doing that.

126
00:06:49,213 --> 00:06:50,303
Like, I, I don't, I don't

127
00:06:50,353 --> 00:06:50,883
Fraser Kelton: Yeah, yeah, that,

128
00:06:51,423 --> 00:06:52,133
Nabeel Hyatt: it's not them.

129
00:06:52,143 --> 00:06:56,953
It just felt like a hiding the ball when
clearly even with code, like the code

130
00:06:56,963 --> 00:06:58,333
generation at a Gemini is quite good.

131
00:06:58,383 --> 00:06:59,743
I just don't understand
why they even did it.

132
00:06:59,873 --> 00:07:02,303
Fraser Kelton: I think if you strip
it away and you look at a comparable

133
00:07:02,313 --> 00:07:07,753
measure of five prompt, five examples
in the prompt, GPT 4 outperforms

134
00:07:07,753 --> 00:07:10,213
it, but it only outperforms it
by a couple of percentage points.

135
00:07:10,433 --> 00:07:12,683
But again, like, I think that
once we get our hands on it, my

136
00:07:12,703 --> 00:07:16,983
guess is that we will see that
this is directionally similar ish.

137
00:07:17,923 --> 00:07:19,983
Nabeel Hyatt: let's finish on this rant
and then I actually want to talk about

138
00:07:19,983 --> 00:07:21,573
the thing I really loved about Gemini.

139
00:07:21,903 --> 00:07:27,343
The part that I think was unfortunate and
I hope no startup takes away from is that

140
00:07:27,463 --> 00:07:30,133
everybody gets excited because there's
an announcement from Google that there's

141
00:07:30,193 --> 00:07:35,593
finally Gemini out and within a few
hours, it just dawns on everybody that,

142
00:07:35,983 --> 00:07:40,853
okay, Gemini is not really here because
it's just pro, we don't have an exact

143
00:07:40,853 --> 00:07:46,568
release date still, this, this graph of
evals is, Cooked in a couple places and

144
00:07:46,568 --> 00:07:51,358
the rest of them are still comparing to
GPT 4 back in March eval tests and only

145
00:07:51,358 --> 00:07:56,788
beat them by three five percent and since
then GPT 4 has gotten a lot better and

146
00:07:56,938 --> 00:07:58,438
By the way, evals don't really matter.

147
00:07:58,538 --> 00:08:05,443
So like that's the negative side the
positive side is So many companies

148
00:08:05,783 --> 00:08:13,473
absolutely fail to show their
product in action in unique and novel

149
00:08:13,473 --> 00:08:14,763
ways that pull the heartstrings.

150
00:08:15,243 --> 00:08:19,933
And I think they, if you haven't, if you
guys are listening to this on the podcast

151
00:08:19,933 --> 00:08:24,163
and you haven't yet watched the Gemini,
Demo videos go on YouTube, you should

152
00:08:24,193 --> 00:08:29,813
take a look there's some wonderful craft
work that is not too pretentious and not

153
00:08:29,813 --> 00:08:34,663
too overblown, but just, in fact, it's
very clean and simple, except for the

154
00:08:34,663 --> 00:08:38,583
YouTuber Roper, Roper video, that's kind
of overblown, but the rest of it is very

155
00:08:38,593 --> 00:08:43,533
simple, good demos of showing various ways
that this product can be put into use that

156
00:08:43,533 --> 00:08:47,133
an average consumer, which might be the
aim of this announcement, is much more

157
00:08:47,133 --> 00:08:51,703
the average consumer You know, or Google
stockholder then it is aimed at engineers.

158
00:08:51,703 --> 00:08:52,883
I mean, didn't you love those demo videos?

159
00:08:52,883 --> 00:08:53,443
You watch them, right?

160
00:08:55,073 --> 00:08:58,613
Fraser Kelton: Oh, yeah, but
I, I don't know if You're

161
00:08:58,613 --> 00:09:00,293
getting me riled up here, man.

162
00:09:00,493 --> 00:09:06,213
I had shared the one demo video where
they show it, the three cup technique

163
00:09:06,213 --> 00:09:09,783
where there's one ball underneath the
cup and then they shuffle the cups around

164
00:09:10,083 --> 00:09:12,363
and it tells them which cup it's under.

165
00:09:12,703 --> 00:09:16,443
Because this, It's a multi modal
model that has been trained from

166
00:09:16,443 --> 00:09:18,423
the start for multi modality.

167
00:09:18,493 --> 00:09:24,113
So, it's accounting for text and image
and, and video and, and audio right

168
00:09:24,113 --> 00:09:28,093
from the, the start of pre training
steps, rather than, you know, kind

169
00:09:28,093 --> 00:09:29,843
of bridging that in after the fact.

170
00:09:30,353 --> 00:09:33,853
And this, this demo video is one of
the best demo videos I've ever seen.

171
00:09:35,173 --> 00:09:35,433
Nabeel Hyatt: Yep.

172
00:09:35,743 --> 00:09:38,803
Fraser Kelton: And then, and then
it comes out that it's all fake!

173
00:09:39,368 --> 00:09:39,398
Nabeel Hyatt: Right.

174
00:09:39,973 --> 00:09:43,483
Fraser Kelton: So in the
demo video, go watch it.

175
00:09:43,493 --> 00:09:44,613
It's, it's remarkable.

176
00:09:44,613 --> 00:09:48,773
There's a a set of hands and some
cups and a ball and the demo says,

177
00:09:48,773 --> 00:09:51,613
okay, now I'm going to put the
cup under here and move it around.

178
00:09:51,913 --> 00:09:52,553
Where is it?

179
00:09:52,853 --> 00:09:57,953
And in, in real time, the Gemini
voice comes back and says, the cups,

180
00:09:58,003 --> 00:10:02,573
I don't know, under the left side,
and the man lifts up the cup on the

181
00:10:02,573 --> 00:10:03,883
left and sure enough, there it is.

182
00:10:04,058 --> 00:10:04,338
Nabeel Hyatt: Right.

183
00:10:04,923 --> 00:10:07,858
Fraser Kelton: And then, People
have discovered shortly thereafter

184
00:10:07,858 --> 00:10:12,668
the fact that this is basically the
equivalent of like a simulated scene

185
00:10:12,678 --> 00:10:17,258
where they had to prompt Engineer
along the way that said, just turns

186
00:10:17,258 --> 00:10:19,168
out to be they fixed it in post.

187
00:10:19,938 --> 00:10:21,068
Nabeel Hyatt: Yeah, that's really

188
00:10:21,198 --> 00:10:23,828
Fraser Kelton: Now, I will compare that,
I will, yeah, I will compare that to

189
00:10:23,838 --> 00:10:27,438
Greg's demo of GPT 4, which was all live.

190
00:10:28,293 --> 00:10:31,103
without any editing and in real time.

191
00:10:31,673 --> 00:10:34,973
And that is the, I think that is the
way that you introduce your products.

192
00:10:36,463 --> 00:10:37,133
It's brave.

193
00:10:37,203 --> 00:10:37,863
It's brave, right?

194
00:10:37,863 --> 00:10:38,493
You're doing it live.

195
00:10:38,493 --> 00:10:39,223
It could fail.

196
00:10:39,283 --> 00:10:42,793
And you, you're owning it because you have
so much confidence in what you've built.

197
00:10:44,543 --> 00:10:47,343
Nabeel Hyatt: uh, I don't,
I can't entirely disagree.

198
00:10:47,743 --> 00:10:50,993
You know, it is as much as I love
the product demo, what I would

199
00:10:50,993 --> 00:10:55,693
have loved was a demo like that
and then a how to behind it.

200
00:10:56,103 --> 00:10:59,723
You know, I think it's okay to make
things that are somewhat polished and

201
00:10:59,723 --> 00:11:04,103
beautiful, but it would be great if it if
it turned out that they reveal the covers.

202
00:11:04,103 --> 00:11:06,093
And by the way, that's no implication on

203
00:11:06,183 --> 00:11:08,283
Fraser Kelton: think, I think,
I think Paula should be.

204
00:11:08,443 --> 00:11:08,793
Nabeel Hyatt: separate

205
00:11:09,118 --> 00:11:09,678
Fraser Kelton: Yeah, yeah.

206
00:11:10,568 --> 00:11:13,988
Polished and beautiful is good, but I
think it has to be grounded in reality.

207
00:11:13,998 --> 00:11:17,418
This is a case where they edited
across two different dimensions, and

208
00:11:17,418 --> 00:11:20,948
people came away with a dramatically
different perspective of what

209
00:11:20,948 --> 00:11:22,268
it is that's actually happening.

210
00:11:23,058 --> 00:11:27,558
And so I think it's unfair to not
allow anybody to use the product, and

211
00:11:27,558 --> 00:11:32,228
then introduce it with a demo video
that basically obfuscates the truth

212
00:11:32,228 --> 00:11:33,458
from two different perspectives.

213
00:11:33,818 --> 00:11:34,928
That's just weird.

214
00:11:35,166 --> 00:11:41,801
I think that this is An excellent
moment for everybody at Google because

215
00:11:41,831 --> 00:11:46,751
they've shipped, or at least they've
partially shipped and I think that

216
00:11:46,881 --> 00:11:49,471
you, they've taken the first step.

217
00:11:49,681 --> 00:11:50,921
No, they've taken the first step.

218
00:11:50,921 --> 00:11:51,591
No, no, no, no, no.

219
00:11:51,591 --> 00:11:53,851
Like, listen, this, the
race was set off a year ago.

220
00:11:53,851 --> 00:11:55,361
They, they did this in a year.

221
00:11:56,081 --> 00:12:00,971
foR a company of their size, this
is, this is not to be, scoffed at.

222
00:12:01,321 --> 00:12:02,801
Think about all of the complexities.

223
00:12:02,841 --> 00:12:05,731
They had to live through smashing
together brain and deep mind.

224
00:12:06,121 --> 00:12:10,251
They had to go and find, like, the
path through all the bureaucracy and

225
00:12:10,251 --> 00:12:14,431
politics to get an aggregate amount of
compute required to be able to do this.

226
00:12:14,731 --> 00:12:17,071
They had to solve all of the
different challenges, both

227
00:12:17,081 --> 00:12:20,111
technically and politically,
within the organization to do this.

228
00:12:20,726 --> 00:12:21,466
And it's out.

229
00:12:21,576 --> 00:12:25,866
And I think that, that itself is
something that should be respected.

230
00:12:25,866 --> 00:12:28,736
And, we can squabble over the evals
and stuff, and the proof will be

231
00:12:28,736 --> 00:12:29,776
when we actually get to use it.

232
00:12:30,336 --> 00:12:32,416
But it looks, it looks directionally good.

233
00:12:32,666 --> 00:12:33,506
And that's something.

234
00:12:33,556 --> 00:12:38,386
You know, I, I feel like I also
did a good job playing your role.

235
00:12:38,386 --> 00:12:41,526
You usually are the one who's
clairvoyant in, in many respects.

236
00:12:41,696 --> 00:12:46,961
And at that dinner, My guess was that
Google was going to come roaring back.

237
00:12:47,056 --> 00:12:47,536
Nabeel Hyatt: your quote.

238
00:12:47,551 --> 00:12:47,901
Fraser Kelton: that.

239
00:12:48,641 --> 00:12:48,971
Yep.

240
00:12:49,201 --> 00:12:55,161
Because they are The best at, at the
technical pieces that have to come

241
00:12:55,161 --> 00:12:57,711
together for training a model like this.

242
00:12:58,081 --> 00:13:01,661
And if you look at some of the stats,
I forget what the stats called, but

243
00:13:01,661 --> 00:13:06,831
for basically the measure of efficiency
when they were training Ultra, I think

244
00:13:06,831 --> 00:13:12,631
they reached some level of like 90%, 97
percent efficiency in the utilization

245
00:13:12,631 --> 00:13:16,561
of their hardware when training Ultra,
which is just a remarkable achievement.

246
00:13:17,041 --> 00:13:20,051
And this is the area where we
should expect them to be great.

247
00:13:20,556 --> 00:13:24,136
And I think they have shown
that they can be great, if not,

248
00:13:24,136 --> 00:13:25,256
you know, on a year's delay.

249
00:13:26,021 --> 00:13:30,131
And then I think the real challenge for
them is going to be how they bring the

250
00:13:30,131 --> 00:13:35,561
great technical piece into their two
products that are now the two front war.

251
00:13:35,611 --> 00:13:39,961
BARD, as you laughed earlier, is a
thing, and then the second one is they're

252
00:13:39,961 --> 00:13:44,461
going to have to find the right way to
integrate these technologies into search.

253
00:13:44,971 --> 00:13:47,841
And that's going to be an
excruciatingly hard challenge because

254
00:13:47,841 --> 00:13:53,021
it's orthogonal to the business
model that is search historically.

255
00:13:53,936 --> 00:13:58,966
Nabeel Hyatt: Look, I, I, I, I say if I
were, I'm not to speak for Demi or Eli

256
00:13:58,966 --> 00:14:01,596
or anybody else over the Google team and
what they're doing, I'm sure they know

257
00:14:01,596 --> 00:14:05,626
a lot more about how to do this than
we do, but, but I do think their role

258
00:14:06,196 --> 00:14:10,966
or their way to fit if I were trying
to navigate this space and I was Google

259
00:14:10,996 --> 00:14:15,636
was to take almost an Apple approach
to this given their scale and size.

260
00:14:15,666 --> 00:14:19,336
And what I mean by that is I, I always
joke people think of Apple as innovative,

261
00:14:19,416 --> 00:14:23,856
and I think of Apple as a last mover
advantage, not first advantage company.

262
00:14:24,236 --> 00:14:27,956
They have had a few moments in their
life where they have been very early,

263
00:14:28,256 --> 00:14:30,926
but in many ways it's taking the
things that are already out there,

264
00:14:30,926 --> 00:14:32,246
that are already somewhat proven.

265
00:14:32,606 --> 00:14:37,286
And then making them so polished
and so well thought through that

266
00:14:37,316 --> 00:14:40,856
you just, they feel like they
fit into your life immediately.

267
00:14:40,876 --> 00:14:44,846
And, you know, they were not the first
to release little notification widgets

268
00:14:44,846 --> 00:14:46,686
on a smartphone that was Android.

269
00:14:47,016 --> 00:14:50,146
They're not the first to do
wireless charging that was Android.

270
00:14:50,566 --> 00:14:54,236
Go way back, they, they, they took a lot
of their early ideas from Xerox PARC.

271
00:14:54,466 --> 00:14:57,966
And so if Google wants to the game
of being last, because it's really

272
00:14:57,966 --> 00:15:00,926
gonna work and work reliably,
there is a game to be played there.

273
00:15:01,206 --> 00:15:03,246
Because I don't think OpenAI
wants to play that game, to be

274
00:15:03,246 --> 00:15:04,396
honest, and you can't play both.

275
00:15:04,406 --> 00:15:08,631
I think right now, in many ways,
OpenAI Playing closer to the Android

276
00:15:08,651 --> 00:15:13,761
or Samsung, if we're going to use
smartphone analogy model where they are

277
00:15:13,771 --> 00:15:15,651
riding the front edge of development.

278
00:15:15,731 --> 00:15:19,291
It drives them crazy if somebody else gets
something out new ahead of them and they

279
00:15:19,291 --> 00:15:21,531
want to play the front edge of the game.

280
00:15:21,801 --> 00:15:24,891
I think both can be successful
strategies as long as the thing that

281
00:15:24,901 --> 00:15:30,496
Google eventually releases As you get
to Ultra is worth the time and energy.

282
00:15:30,526 --> 00:15:33,916
That's the, you know, the, like,
it's worth weight that that's the

283
00:15:33,916 --> 00:15:36,466
thing that will be left to find out.

284
00:15:37,736 --> 00:15:38,631
Fraser Kelton: We shall see.

285
00:15:38,631 --> 00:15:40,356
We shall see.

286
00:15:40,516 --> 00:15:41,716
Nabeel Hyatt: look, not, it's hard.

287
00:15:42,016 --> 00:15:44,471
Of course, the cup example is tough.

288
00:15:44,951 --> 00:15:47,651
You know, these, these
prompts are hard to shape.

289
00:15:47,861 --> 00:15:50,681
It's hard to get the little alien
inside my computer to understand

290
00:15:50,831 --> 00:15:51,971
that I'm playing a Cup game.

291
00:15:52,471 --> 00:15:58,731
Fraser Kelton: Issue with the cup thing is
that they imply that they lead the viewer

292
00:15:58,731 --> 00:16:00,481
to believe that there's zero prompting.

293
00:16:00,481 --> 00:16:02,071
It's not that prompting's hard.

294
00:16:02,546 --> 00:16:07,366
The way that the video is presented
suggests that there's zero prompting and

295
00:16:07,366 --> 00:16:14,356
that there's this real time multimodal
model watching you and, and the cups

296
00:16:14,376 --> 00:16:16,826
and inferring with real reasoning

297
00:16:16,896 --> 00:16:22,486
and there's somewhat complex prompting
happening at each step behind the

298
00:16:22,486 --> 00:16:25,656
scenes, which is what I think has,
has caused everybody to be really

299
00:16:25,656 --> 00:16:29,096
disappointed in, in a decision to do that.

300
00:16:29,431 --> 00:16:33,421
Nabeel Hyatt: The last thing I'd say
on Gemini, is that, is a lot of this

301
00:16:33,441 --> 00:16:36,031
consternation would have been solved.

302
00:16:36,531 --> 00:16:40,771
If they would have released APIs for
developers to build with at the same time.

303
00:16:41,621 --> 00:16:45,311
And I think, I think we've, supposedly
gonna come out December 13th.

304
00:16:45,311 --> 00:16:47,111
I don't know if Ultra's
gonna be involved in that.

305
00:16:47,631 --> 00:16:53,901
But in a world of AI movement that's
five, seven days from now, I mean

306
00:16:55,031 --> 00:16:59,861
op open AI fires a CEO and goes
through a, a coup attempt then gets

307
00:16:59,861 --> 00:17:01,271
back a CEO in that time period.

308
00:17:01,271 --> 00:17:02,711
Like a lot, a lot happens in five days.

309
00:17:04,101 --> 00:17:05,291
Fraser Kelton: lot happens in five days.

310
00:17:05,971 --> 00:17:10,301
Nabeel Hyatt: and so, like, I'm sure
this was PR oriented, they wanted

311
00:17:10,301 --> 00:17:14,791
people to watch a Mark Roper video and
so on and so forth before developers

312
00:17:14,791 --> 00:17:16,251
had control of the narrative.

313
00:17:16,501 --> 00:17:21,791
But it's really unfortunate on,
because I think it creates a sense

314
00:17:22,101 --> 00:17:24,611
of doubt when it shouldn't be.

315
00:17:24,651 --> 00:17:28,661
It should just be high fives,
hand clapping, and playground.

316
00:17:29,051 --> 00:17:32,441
And I think that was a little bit
of a PR mishap, but we'll, we'll

317
00:17:32,441 --> 00:17:34,246
see what happens in, in seven days.

318
00:17:35,701 --> 00:17:36,001
Fraser Kelton: Yep.

319
00:17:36,321 --> 00:17:36,691
Yep.

320
00:17:37,251 --> 00:17:39,251
And anyway, prompt, prompting is hard.

321
00:17:40,041 --> 00:17:46,331
We, we talked last time about efforts in
ChatGPT to simplify the complexity and

322
00:17:46,331 --> 00:17:51,241
ambiguity of prompting specifically with
DALI, where they want to take the, the

323
00:17:51,241 --> 00:17:55,931
three words that somebody who's unfamiliar
or lazy with their, their directions wants

324
00:17:55,931 --> 00:18:00,641
to do and how, if you're a power user
such as yourself, it's just suboptimal.

325
00:18:02,036 --> 00:18:02,856
Nabeel Hyatt: Yeah, it's,

326
00:18:02,936 --> 00:18:03,426
Fraser Kelton: What, what

327
00:18:03,746 --> 00:18:07,436
Nabeel Hyatt: I saw my great example
of that this week because I'm going

328
00:18:07,436 --> 00:18:15,286
to keep banging the drum that I think
Prompt Engineering is a real skill and

329
00:18:15,286 --> 00:18:19,106
will be a career for quite some time
and that actually Prompt Engineering

330
00:18:19,566 --> 00:18:22,566
is going to become more of a language.

331
00:18:23,136 --> 00:18:27,116
Before it eventually gets abstracted out,
but our ability to totally abstract it

332
00:18:27,116 --> 00:18:29,786
out while we're still trying to figure
out what these non deterministic models

333
00:18:29,786 --> 00:18:34,646
can actually do is very, is very far
away, maybe years and years away before

334
00:18:34,646 --> 00:18:36,246
we can build these systems on top of them.

335
00:18:36,706 --> 00:18:40,276
I got handed a wonderful
example of this today, I sent

336
00:18:40,276 --> 00:18:42,026
it your way which is Anthropic.

337
00:18:42,671 --> 00:18:46,481
Which we are investors in by disc
full disclosure to everybody.

338
00:18:46,781 --> 00:18:50,401
Anthropic has a competitive model
to OpenAI and Gemini called Claude.

339
00:18:51,071 --> 00:18:56,141
And there is a well known research
problem and execution problem

340
00:18:56,161 --> 00:18:57,961
in these long context windows.

341
00:18:58,351 --> 00:19:03,151
of AI where I'm asking it to, for
instance, look at an entire PDF or look

342
00:19:03,151 --> 00:19:08,141
at a long chat and find some phrase
or find some word inside of that, did

343
00:19:08,141 --> 00:19:11,671
Sam talk about the beach or not, or
what's the best cooking technique?

344
00:19:11,671 --> 00:19:15,196
And it turns out that if it's mentioned
in the beginning of a doc, Or at the

345
00:19:15,196 --> 00:19:20,236
end of a doc every model, all of these
LLMs show that they can find information

346
00:19:20,236 --> 00:19:24,296
at the beginning of a doc and at
the end of the doc faster and more

347
00:19:24,296 --> 00:19:26,226
reliably than in the middle of the doc.

348
00:19:26,226 --> 00:19:29,396
The middle, it's the missing middle
it just sometimes misses stuff.

349
00:19:29,911 --> 00:19:34,251
Well, this has been a quote unquote
known thing, of which people have been

350
00:19:34,251 --> 00:19:37,661
trying to do all kinds of different
engineering techniques chunking the

351
00:19:37,661 --> 00:19:41,481
data into smaller bits, and then
there's like comparison evals against

352
00:19:41,481 --> 00:19:44,491
different models at different times,
and how they perform on the missing

353
00:19:44,501 --> 00:19:45,471
middle, and so on and so forth.

354
00:19:45,971 --> 00:19:50,381
And then it turns out that Claude
releases a paper today called Claude 2.

355
00:19:50,381 --> 00:19:55,081
1 Prompting, that says, well,
what did it say, Fraser?

356
00:19:55,601 --> 00:20:00,476
What's the crazy, deep Engineering
technique that, that scientists

357
00:20:00,476 --> 00:20:04,786
have figured out in order to finally
unlock moving from 23 percent

358
00:20:04,786 --> 00:20:08,896
missing middle accuracy up to 97
percent missing middle accuracy.

359
00:20:10,496 --> 00:20:16,626
Fraser Kelton: mean, they add a
line to the prompt that says here

360
00:20:16,626 --> 00:20:21,046
is the most relevant sentence in the
context, which basically nudges the

361
00:20:21,046 --> 00:20:27,796
prompt to go and pull out the relevant
sentence for the question at hand.

362
00:20:28,546 --> 00:20:29,696
And that's the bump,

363
00:20:30,676 --> 00:20:30,856
Nabeel Hyatt: Yeah.

364
00:20:30,856 --> 00:20:33,046
I mean, that's insane.

365
00:20:33,096 --> 00:20:36,286
This afternoon, I'm going to do
some work and figure out whether

366
00:20:36,286 --> 00:20:38,036
this works in GPT 4 as well.

367
00:20:38,036 --> 00:20:40,416
I didn't have a chance to come up,
but, but it'd be really interesting.

368
00:20:40,751 --> 00:20:43,841
If both of the results, don't you
think, Fraser, would be interesting?

369
00:20:43,841 --> 00:20:49,421
Like, if that phrasing does work in
GPT 4 as well then it's like, oh, you

370
00:20:49,421 --> 00:20:52,651
just figured out a new incantation,
kind of like we found out that if

371
00:20:52,651 --> 00:20:56,441
you, you tell a model you're going
to tip it to do something, I'll give

372
00:20:56,441 --> 00:20:58,081
you 20 if you answer this question.

373
00:20:58,151 --> 00:21:00,781
They, it tends to perform better in
that question, even though, of course,

374
00:21:00,781 --> 00:21:02,291
you're not giving the model 20.

375
00:21:02,681 --> 00:21:03,951
Another crazy incantation.

376
00:21:04,476 --> 00:21:08,426
And then if, so one, if it worked,
that's interesting, and it tells us

377
00:21:08,466 --> 00:21:12,556
more, a little tip into the language
of how to use these models for large

378
00:21:12,556 --> 00:21:16,326
context windows, which is particularly
valuable for Claude, because it

379
00:21:16,336 --> 00:21:18,176
has such a large context window.

380
00:21:18,176 --> 00:21:20,026
You can just put lots and
lots of text in there.

381
00:21:21,326 --> 00:21:24,596
If it doesn't work in other models,
that's even more interesting, right?

382
00:21:25,476 --> 00:21:29,686
Now, for all these companies that are
trying to say, don't worry, I'm building

383
00:21:30,006 --> 00:21:33,636
middleware dev tools that let you switch
in and out models arbitrarily, with

384
00:21:33,666 --> 00:21:35,466
the, like, like they're all the same.

385
00:21:36,006 --> 00:21:36,686
That they're not.

386
00:21:37,141 --> 00:21:40,841
Fraser Kelton: I would be so
surprised if they're the same today.

387
00:21:40,851 --> 00:21:43,681
And the difference is only
going to grow over time.

388
00:21:44,681 --> 00:21:47,711
There's a whole bunch of
different things going on here.

389
00:21:47,721 --> 00:21:51,801
This is a quote unquote eval
called Needle in the Haystack.

390
00:21:51,966 --> 00:21:56,346
And I think that, yet again, this is
a situation where the eval doesn't

391
00:21:56,356 --> 00:22:00,456
measure anything proximately close
to what happens in, in production

392
00:22:00,456 --> 00:22:05,606
for people who are building products,
right, is if you, if you insert into

393
00:22:05,606 --> 00:22:09,136
the middle of some financial set of
documents, a single sentence that

394
00:22:09,136 --> 00:22:14,126
says Dolores Park is the best place
to have a drink in San Francisco and

395
00:22:14,126 --> 00:22:15,586
then the model can't find it, right.

396
00:22:16,076 --> 00:22:20,306
I'm not sure that that is reflective
of any real world problem that

397
00:22:20,306 --> 00:22:21,456
people are trying to solve with this.

398
00:22:21,496 --> 00:22:27,946
The other thing that is so interesting
here is the Anthropic model they

399
00:22:27,946 --> 00:22:31,716
hypothesized performed poorly when
people were running it through that,

400
00:22:31,736 --> 00:22:38,116
that eval because they've trained their
models to cut down on inaccuracies

401
00:22:38,146 --> 00:22:41,356
specifically for these types of use cases.

402
00:22:41,671 --> 00:22:42,001
Right.

403
00:22:42,001 --> 00:22:46,961
And so they basically have trained the
model to say, okay, if something feels

404
00:22:46,961 --> 00:22:51,441
completely orthogonal from the rest of
the documents, it's probably not something

405
00:22:51,441 --> 00:22:53,431
that's, that's important and or accurate.

406
00:22:53,431 --> 00:22:54,561
It's probably not even accurate.

407
00:22:54,561 --> 00:22:56,061
So, so just ignore that.

408
00:22:56,531 --> 00:22:56,831
Right.

409
00:22:56,851 --> 00:23:01,161
And then the eval is basically testing for
the model performance to do exactly that.

410
00:23:01,314 --> 00:23:04,014
Nabeel Hyatt: Yeah, but I want to get back
to the point that I wanted to make, which

411
00:23:04,264 --> 00:23:11,574
this is six words that you put into a
prompt if you were trying to do long text

412
00:23:11,574 --> 00:23:18,554
retrieval, text from a long context window
that, that does boost performance and.

413
00:23:19,609 --> 00:23:24,899
I don't know, it just, it tells
me how naive we are collectively

414
00:23:24,919 --> 00:23:26,399
about how to use these models.

415
00:23:27,809 --> 00:23:33,046
Um, Emma, who's an AI hacker in
residence for us, she did a benchmark

416
00:23:33,056 --> 00:23:38,096
on some internal tools that she was
using on Glaive versus GPT and she

417
00:23:38,096 --> 00:23:42,696
found that without prompt engineering,
Glaive did better than GPT 4 probably

418
00:23:42,696 --> 00:23:46,096
because it's trained only on highly
quality synthetic data and so on and

419
00:23:46,096 --> 00:23:50,771
so forth, but that if you added the
sentence, You're a well known historian

420
00:23:50,961 --> 00:23:58,031
to the prompt for both Glaive and
GPT that then 4 suddenly did better.

421
00:23:59,031 --> 00:24:03,511
And it's just another good testament
to you just need to find the magic five

422
00:24:03,741 --> 00:24:09,291
incantation words to suddenly make your
business be able to move into prod.

423
00:24:09,291 --> 00:24:10,526
That's

424
00:24:10,571 --> 00:24:11,731
Fraser Kelton: that is so crazy.

425
00:24:11,731 --> 00:24:16,261
They could just even try to internalize
that in the brittleness of these models.

426
00:24:16,271 --> 00:24:20,861
You're going to have, you are,
you're a well known historian and

427
00:24:20,861 --> 00:24:22,221
then it finally outperforms it.

428
00:24:22,751 --> 00:24:25,641
I don't know how this gets solved,
like, other than at the system level.

429
00:24:25,821 --> 00:24:28,701
Nabeel Hyatt: But, in the very early
days of video games, you worked at the

430
00:24:28,701 --> 00:24:31,041
assembly level to make a video game.

431
00:24:31,041 --> 00:24:34,851
In the early days of computer
graphics, before we got to

432
00:24:34,901 --> 00:24:37,111
engines, we had to work in code.

433
00:24:37,611 --> 00:24:41,621
And we will eventually get
to lots of GUIs and engines.

434
00:24:41,621 --> 00:24:45,081
And we've actually talked before
about how prompt engineering is not

435
00:24:45,091 --> 00:24:47,901
how every average user to use these
products and people are bad at English.

436
00:24:48,336 --> 00:24:51,566
But at the same time, if you need
performance, you need to be at bare

437
00:24:51,566 --> 00:24:56,186
metal as close to the model as possible
for probably a little while until

438
00:24:56,376 --> 00:24:59,376
everything really, really works.

439
00:24:59,706 --> 00:25:04,196
And at that point, when it's automatic
when we've made our 50th first person

440
00:25:04,196 --> 00:25:07,176
shooter, That's in production and making
hundreds of millions of dollars a year.

441
00:25:07,406 --> 00:25:09,886
Then we can talk about making an engine
for making first person shooters.

442
00:25:09,886 --> 00:25:13,306
And when we get in game parlance,
you get Unreal and and you get

443
00:25:13,306 --> 00:25:14,546
Unity and so on and so forth.

444
00:25:14,546 --> 00:25:19,076
But it feels like we are still in,
you know, how was Pac Man built?

445
00:25:20,286 --> 00:25:22,406
Fraser Kelton: I don't, I don't want
to open up this can of worms, but

446
00:25:22,446 --> 00:25:25,186
don't you think that is a measure of.

447
00:25:25,251 --> 00:25:28,901
The model's capabilities
not being strong enough.

448
00:25:28,951 --> 00:25:29,821
Nabeel Hyatt: We know that.

449
00:25:30,251 --> 00:25:36,771
Just like in, like, early programming
days, you were wrangling with

450
00:25:36,771 --> 00:25:38,901
the amount of memory on computer.

451
00:25:39,111 --> 00:25:41,911
You've got a thousand twenty four
bytes of memory, and you're just

452
00:25:41,921 --> 00:25:45,091
trying to make a spreadsheet work in
this tiny little bit of memory, and

453
00:25:45,091 --> 00:25:51,071
you need every little squeezing bit of
thing just to make it operate, right?

454
00:25:51,181 --> 00:25:56,471
And isn't about speed the way it often
was back then but it's still about

455
00:25:57,071 --> 00:25:58,581
whether the job can be done well or not.

456
00:25:59,321 --> 00:26:03,211
And, and yeah, we'll need to be very
close to bare metal, until all these

457
00:26:03,211 --> 00:26:09,231
things run perfectly all the time, and be
about efficiency and cost and abstraction

458
00:26:09,346 --> 00:26:10,226
Fraser Kelton: what Yep

459
00:26:10,701 --> 00:26:11,395
Nabeel Hyatt: the rest of that stuff.

460
00:26:11,525 --> 00:26:17,295
If five words for your specific use
case going to increase performance,

461
00:26:17,585 --> 00:26:23,132
then I don't know if I'm Fidelity or
Procter Gamble or Figma, or InstaWork

462
00:26:23,219 --> 00:26:28,749
or another startup , like, I don't know
that I'm willing to take the future of

463
00:26:28,749 --> 00:26:36,219
my business's effectiveness in AI, which
could twist and turn on five words,

464
00:26:37,149 --> 00:26:37,599
Fraser Kelton: right.

465
00:26:37,839 --> 00:26:38,319
Yeah.

466
00:26:38,469 --> 00:26:40,879
Nabeel Hyatt: And who's going
to figure out those five words

467
00:26:40,879 --> 00:26:41,949
for your specific business?

468
00:26:41,959 --> 00:26:46,649
It's certainly not going to be
some random middleware company.

469
00:26:46,669 --> 00:26:49,329
It's going to be you because you
care about your company and you've

470
00:26:49,699 --> 00:26:52,552
hacked away at it or had a prompt
engineer who's hacking away at it.

471
00:26:52,552 --> 00:26:57,712
You've really worked it to try and
figure out how to wrangle this alien

472
00:26:57,892 --> 00:26:59,272
to do the work that you want it to do.

473
00:27:00,280 --> 00:27:04,430
Fraser Kelton: The point here is that
the brittleness of these models today

474
00:27:04,450 --> 00:27:08,020
across different use cases suggests that
you're going to want to have people,

475
00:27:08,040 --> 00:27:10,178
quote unquote, like, working at the metal.

476
00:27:10,178 --> 00:27:10,926
Yep.

477
00:27:11,240 --> 00:27:14,480
Nabeel Hyatt: analogy I would use is,
in the really early days of the web,

478
00:27:14,712 --> 00:27:20,472
there was almost immediately
A bunch of WYSIWYG web page

479
00:27:20,472 --> 00:27:22,242
developer software companies.

480
00:27:22,282 --> 00:27:27,582
There were 30 startups that were like, you
don't have to learn CSS and HTML just use

481
00:27:27,582 --> 00:27:32,642
our little product, and you can get your
web page out without tweaking it at all.

482
00:27:33,072 --> 00:27:36,647
And, you know, if we fast
forward 10 years, of course,

483
00:27:36,647 --> 00:27:37,777
there's many of those companies.

484
00:27:37,787 --> 00:27:42,002
Today, there's Squarespace and
webflow, a bunch of these companies

485
00:27:42,002 --> 00:27:45,842
that are helping everybody from a
restaurant up the street all the

486
00:27:45,842 --> 00:27:48,262
way to complex enterprise websites.

487
00:27:48,572 --> 00:27:57,352
But in the early days, As a good example,
prior to CSS, the way that you laid

488
00:27:57,362 --> 00:28:01,412
things out on a webpage, so the way I got
something to show up on the right hand

489
00:28:01,412 --> 00:28:05,912
side of a webpage versus the left hand
side of a webpage, was to use a kludge

490
00:28:06,472 --> 00:28:11,222
which is to build a table, kind of like
a spreadsheet on that webpage in HTML,

491
00:28:11,702 --> 00:28:16,412
and then, in one of the cells on the
right hand put my logo so it's on the

492
00:28:16,412 --> 00:28:22,792
right, and then make the cells of that
spreadsheet And it's a, for me, it feels

493
00:28:22,792 --> 00:28:29,812
like we are way more in that land than
we are in, in WYSIWYG abstraction land.

494
00:28:29,862 --> 00:28:33,632
And so the whole first wave,
the whole first couple of years.

495
00:28:33,648 --> 00:28:39,227
Of WYSIWYG website builder companies all
went out of business very, very quickly.

496
00:28:40,967 --> 00:28:41,887
What, what happened there?

497
00:28:41,887 --> 00:28:43,547
If we're going to use that
analogy, what happened there?

498
00:28:44,247 --> 00:28:47,737
WhAt would be the business, if you
wanted to help a million companies

499
00:28:48,737 --> 00:28:50,697
build their first LLM applications.

500
00:28:51,487 --> 00:28:55,667
And the contention is that it's
not the time to build the square

501
00:28:55,667 --> 00:28:59,417
space of the space, uh, which I'm
not, by the way, you know, this

502
00:28:59,417 --> 00:29:01,471
is us just chatting on a podcast

503
00:29:02,481 --> 00:29:03,805
.
A founder could walk in tomorrow.

504
00:29:04,250 --> 00:29:08,720
And pitch and pitch the most beautiful
wonder idea for Squarespace for AI

505
00:29:08,720 --> 00:29:11,230
and, and just prove you totally wrong.

506
00:29:11,260 --> 00:29:12,420
And that's the joy of this process,

507
00:29:12,675 --> 00:29:14,595
Fraser Kelton: That's,
that's what this rule so fun.

508
00:29:14,880 --> 00:29:16,010
Nabeel Hyatt: Yeah, exactly.

509
00:29:16,230 --> 00:29:19,240
So strong, strong convictions,
really loosely held.

510
00:29:19,540 --> 00:29:22,660
But, actually, do you
believe in my analogy?

511
00:29:22,660 --> 00:29:25,000
Do you think that's an apt analogy
or do you think I'm full of it?

512
00:29:25,550 --> 00:29:26,820
Fraser Kelton: No, I don't
think you're full of it.

513
00:29:26,960 --> 00:29:33,010
So, if I understand what's happened in the
Anthropic case, it is The way that they

514
00:29:33,020 --> 00:29:38,890
have tried to nudge the model to improve
performance has then resulted in some

515
00:29:38,950 --> 00:29:45,450
wonky behavior that you can then nudge it
over that hurdle with five magic words.

516
00:29:45,870 --> 00:29:47,100
And what does that say to me?

517
00:29:47,100 --> 00:29:51,830
That, that says to me that
there's probably a solution that

518
00:29:51,840 --> 00:29:54,310
happens at the system level.

519
00:29:54,350 --> 00:29:57,980
If you think about how this may mature
why would they want their customers

520
00:29:57,980 --> 00:29:59,030
to ever have to think about that?

521
00:29:59,030 --> 00:30:03,900
They'll, they'll find ways to absorb
the solution or abstract the solution

522
00:30:03,970 --> 00:30:05,765
for use cases where it makes sense.

523
00:30:06,007 --> 00:30:07,319
Nabeel Hyatt: Yeah, but I
don't have time for that.

524
00:30:07,319 --> 00:30:10,045
I'm a founder that wants
first mover advantage.

525
00:30:10,248 --> 00:30:14,509
Or, My boss me that I need to have an
AI strategy and I need to, I need to

526
00:30:14,509 --> 00:30:17,819
launch next month and it can't, it's
got to get out of demo land cause I've

527
00:30:17,819 --> 00:30:19,399
got an earnings report next quarter.

528
00:30:19,449 --> 00:30:23,579
Fraser Kelton: This, this is
why that that person is having.

529
00:30:23,914 --> 00:30:25,384
Random success.

530
00:30:25,394 --> 00:30:27,684
Sometimes they're succeeding,
sometimes they're failing.

531
00:30:27,754 --> 00:30:30,444
And sometimes they come back to
the drawing board with an entirely

532
00:30:30,444 --> 00:30:31,974
new approach one month later.

533
00:30:32,014 --> 00:30:33,174
We've seen that a lot.

534
00:30:33,789 --> 00:30:34,549
Nabeel Hyatt: That's very true.

535
00:30:35,109 --> 00:30:36,459
I do wonder.

536
00:30:37,059 --> 00:30:41,309
If Procter and Gamble and, and Fidelity
and JPMorgan and every other company

537
00:30:41,309 --> 00:30:43,469
is trying to figure out how to use AI.

538
00:30:44,549 --> 00:30:47,439
If I just think about the web, the
web analogy for a second, and you

539
00:30:47,439 --> 00:30:50,959
don't want to overstretch any analogy
of course, but the really effective

540
00:30:50,959 --> 00:30:54,059
companies in that first wave for
helping to bring everybody onto the

541
00:30:54,059 --> 00:30:59,769
web were kind of a mixture of tools
companies slash consulting companies.

542
00:31:00,659 --> 00:31:01,159
Fraser Kelton: Yeah,

543
00:31:01,302 --> 00:31:08,070
Nabeel Hyatt: It was scient and viant
and Razorfish that, you go pay them

544
00:31:08,080 --> 00:31:13,480
hundreds of thousands of dollars
and they would build time magazine.

545
00:31:13,710 --> 00:31:17,800
com for the first time, these
kind of mixture of design agency,

546
00:31:17,980 --> 00:31:20,700
software engineering, and then
they ended up with internal tool

547
00:31:20,700 --> 00:31:22,340
stacks that they knew how to use.

548
00:31:22,790 --> 00:31:25,270
I think there's an analogy to

549
00:31:25,351 --> 00:31:25,911
Fraser Kelton: Oh, hell, yeah,

550
00:31:26,901 --> 00:31:27,631
I mean, yeah.

551
00:31:27,751 --> 00:31:31,911
There's a reason why OpenAI has, I
forget, I'm going to get the names

552
00:31:31,911 --> 00:31:36,271
wrong here, but has a Keystone
partnership with Bain and Anthropic

553
00:31:36,271 --> 00:31:38,651
has a Keystone partnership with BCG.

554
00:31:39,056 --> 00:31:43,566
Is these are footsie things to bring
into the enterprise, as we've seen.

555
00:31:43,586 --> 00:31:45,986
Five words makes the difference between
something that looks horrible and

556
00:31:45,986 --> 00:31:49,666
something that would be delightful in
production, and there has to be people

557
00:31:49,666 --> 00:31:55,166
who can help you navigate that, uh, as the
world is changing underneath your feet.

558
00:31:55,821 --> 00:31:56,401
Three months.

559
00:31:56,481 --> 00:31:56,921
No.

560
00:31:57,646 --> 00:32:03,136
Nabeel Hyatt: Well, the contention
is right that, Razorfish and Scient

561
00:32:03,156 --> 00:32:07,336
and Vine were net new org, yes, they
were consulting organizations that

562
00:32:07,664 --> 00:32:11,724
rhyme with Bain in the way that they
actually but Bain is old school.

563
00:32:12,194 --> 00:32:15,714
Are there really great AI
implementation engineers waiting

564
00:32:15,714 --> 00:32:17,744
at Bain to take you out to market?

565
00:32:17,754 --> 00:32:19,164
Absolutely not, I would guess.

566
00:32:19,724 --> 00:32:27,704
I, I suspected it's a net, that there's
an opportunity for a net new company to be

567
00:32:27,704 --> 00:32:34,704
filled with people who like to implement,
who will help take these tools, which seem

568
00:32:34,724 --> 00:32:37,354
maybe very easy to stand up very quickly.

569
00:32:37,354 --> 00:32:40,564
I can just go to a prompt and type
things in, but I think are probably

570
00:32:40,634 --> 00:32:44,184
more complicated and people will find
are more complicated than they think.

571
00:32:45,004 --> 00:32:46,764
To actually implement and get live.

572
00:32:46,804 --> 00:32:48,864
And that's why I like the HTML analogy.

573
00:32:48,864 --> 00:32:51,714
It's incredibly simple to
build your first HTML page.

574
00:32:52,014 --> 00:32:54,254
But then, and it feels
like anyone can do it.

575
00:32:54,554 --> 00:32:57,294
But actually trying to
run the NewYorkTimes.

576
00:32:57,294 --> 00:33:02,204
com you know, is another whole
order of magnitude more difficult.

577
00:33:02,224 --> 00:33:05,044
And especially in the early days
where people didn't really know

578
00:33:05,044 --> 00:33:06,714
web and how to do web development.

579
00:33:07,084 --> 00:33:10,314
You needed a set of people that were
your launch team and stood up the

580
00:33:10,314 --> 00:33:12,224
internet, you know, website by website.

581
00:33:12,604 --> 00:33:16,204
I think there's a little bit of that
that probably goes on and I just

582
00:33:16,204 --> 00:33:20,874
don't think it's going to be McKinsey
or Bain or the folks that have,

583
00:33:21,429 --> 00:33:25,819
Really very little of this specific
type of DNA, but I could be wrong.

584
00:33:26,674 --> 00:33:26,974
Fraser Kelton: Yeah.

585
00:33:27,024 --> 00:33:31,824
People who did it back in the day for
transitioning people onto the internet.

586
00:33:32,419 --> 00:33:36,609
Did they do it through just specialized
know how, or did they build tools

587
00:33:36,609 --> 00:33:41,209
and platforms that allowed them to,
to simplify the task for others?

588
00:33:41,504 --> 00:33:43,864
Nabeel Hyatt: Like anything you start
out making a thing and then you're

589
00:33:43,864 --> 00:33:47,064
like, once you've done it two or three
times, engineers can't help themselves.

590
00:33:47,484 --> 00:33:50,984
And so you start to build efficient

591
00:33:51,124 --> 00:33:54,794
Fraser Kelton: is it, but that a, so, but
are we back to this is, there actually

592
00:33:54,794 --> 00:33:59,144
is a middleware company, like a tool
that's going to start from a consultancy

593
00:33:59,144 --> 00:34:00,614
type perspective and then get built out?

594
00:34:02,144 --> 00:34:05,144
And then is your, your issue
with the tool startups?

595
00:34:05,144 --> 00:34:07,874
Just the fact that they're not
going to market appropriately.

596
00:34:08,874 --> 00:34:09,904
Nabeel Hyatt: That's a good pushback.

597
00:34:09,924 --> 00:34:10,884
It might be.

598
00:34:11,214 --> 00:34:14,544
I mean, we'll, none of us know, we'll see
how this all plays out, but yeah, maybe

599
00:34:14,544 --> 00:34:16,414
the right way, it's not what VCs want.

600
00:34:16,434 --> 00:34:18,654
Hey, why don't you hire more
implementation engineers?

601
00:34:18,834 --> 00:34:22,464
It's not what VC on a panel would be.

602
00:34:22,464 --> 00:34:23,704
They'd be like, no, no humans.

603
00:34:23,724 --> 00:34:25,244
The AI should write itself.

604
00:34:25,734 --> 00:34:32,004
But for where we are on the technology
side it might be that the right answer

605
00:34:32,004 --> 00:34:34,194
for the next 12 to 18 months is.

606
00:34:34,824 --> 00:34:38,444
You have a whole bunch of
implementation engineers that are

607
00:34:38,444 --> 00:34:44,382
script monkey that know all of the
unique folklore about how to wrangle

608
00:34:44,382 --> 00:34:45,832
these models in the right direction.

609
00:34:45,882 --> 00:34:49,532
So you're still selling your tool
set, but you're selling your tool set

610
00:34:49,582 --> 00:34:53,332
along with a handful of implementation
engineers and a maintenance contract.

611
00:34:53,942 --> 00:34:59,242
And I know that that, that breaks a lot
of the purity software that we would

612
00:34:59,242 --> 00:35:03,972
all love for engineering to be, but
it might be the right thing for, for

613
00:35:03,972 --> 00:35:05,902
this particular stage that we're in.

614
00:35:06,197 --> 00:35:09,107
Fraser Kelton: Could be . You know,
going back to the start of the API,

615
00:35:09,427 --> 00:35:12,367
there were two people, a guy named
Boris and a guy named Andrew at OpenAI

616
00:35:12,867 --> 00:35:17,617
who were prompt wizards, like they
just knew how to, to construct and

617
00:35:17,617 --> 00:35:18,967
orchestrate these things in a way.

618
00:35:18,967 --> 00:35:20,427
And that's what, that's what they did.

619
00:35:20,487 --> 00:35:24,177
They ran around to the implementations
that seemed most interesting and then

620
00:35:24,177 --> 00:35:29,147
helped them sand off the rough edges
to see if it was a path to production.

621
00:35:29,147 --> 00:35:34,112
And in many cases, They could nudge them
there, whereas as few, few people could.

622
00:35:34,642 --> 00:35:36,132
Nabeel Hyatt: Boris is a
great name for a startup.

623
00:35:37,132 --> 00:35:38,472
Fraser Kelton: Yeah, he is remarkable.

624
00:35:38,532 --> 00:35:40,682
He himself could be a startup.

625
00:35:41,072 --> 00:35:44,522
So you don't think that these things
get abstracted the other way, where

626
00:35:44,522 --> 00:35:49,532
they get pulled down into the actual
model level, and that people aren't

627
00:35:49,532 --> 00:35:51,952
interacting with any of this above that.

628
00:35:52,022 --> 00:35:54,782
And, and it kind of ties back
to the Gemini thing, right?

629
00:35:54,902 --> 00:35:56,272
Nabeel Hyatt: Oh, I think
that's a very good point.

630
00:35:56,292 --> 00:35:58,332
Very likely that that happens in parallel.

631
00:35:58,992 --> 00:36:04,922
And, technology tends to go through
ages of entropy and de entropy.

632
00:36:05,462 --> 00:36:11,122
We all love, especially as engineers,
we love de entropy, we love simplifying

633
00:36:11,122 --> 00:36:16,592
everything, cleaning it up getting rid
of the noise from signal bringing it

634
00:36:16,592 --> 00:36:18,592
all down into something that works.

635
00:36:18,862 --> 00:36:23,972
But when things are not working fully
you can't jump three steps ahead, you

636
00:36:24,292 --> 00:36:25,922
have to go through a phase of entropy.

637
00:36:26,102 --> 00:36:29,142
It's why I don't get nervous
about One more model launching

638
00:36:29,142 --> 00:36:30,622
or one more startup launching.

639
00:36:30,872 --> 00:36:35,422
We need as many shots on goal
and bets to move this technology

640
00:36:35,422 --> 00:36:36,882
forward as quickly as possible.

641
00:36:37,382 --> 00:36:42,462
Things that are trying to make a promise
of de entropy too quickly Just feel

642
00:36:42,462 --> 00:36:46,403
incongruous to me when the goal is
solve the problem reliably and, and

643
00:36:46,623 --> 00:36:48,783
we're still not at reliable solution.

644
00:36:49,183 --> 00:36:52,743
And so my is that reliable solution
is going to get way more complicated

645
00:36:52,743 --> 00:36:53,833
before it's going to get easier.

646
00:36:54,828 --> 00:36:57,568
Fraser Kelton: Boy, we wrestled with this
one, but that one feels really right.

647
00:36:58,103 --> 00:37:02,103
It's going to get more complicated
in every direction because we are

648
00:37:02,103 --> 00:37:07,863
not at the reliability required for
consistent value in many use cases.

649
00:37:08,353 --> 00:37:13,303
And like, why bother adding abstractions
of simplicity if you say it's

650
00:37:13,303 --> 00:37:14,323
still not going to be good enough?

651
00:37:15,093 --> 00:37:15,863
Nabeel Hyatt: Yeah, exactly.

652
00:37:15,873 --> 00:37:18,783
Fraser Kelton: lot easier for you to get
something than broken into production.

653
00:37:19,813 --> 00:37:21,423
Nabeel Hyatt: That's
what, that's the headline.

654
00:37:21,713 --> 00:37:24,117
Why are we making it easier to
get broken things into production?

655
00:37:24,337 --> 00:37:26,523
or, we could just fix it
with marketing Frazier.

656
00:37:26,895 --> 00:37:31,189
Fraser Kelton: No, man, this is like,
I'm listening to you talk about Gemini

657
00:37:31,219 --> 00:37:35,562
and then like nudge me and I don't, they,
they misrepresented what the product is.

658
00:37:36,562 --> 00:37:38,422
Nabeel Hyatt: you, you're
not reacting to the evals.

659
00:37:38,482 --> 00:37:39,892
You're reacting to the demo video.

660
00:37:39,922 --> 00:37:40,312
Fraser Kelton: That's right.

661
00:37:40,342 --> 00:37:42,632
Actually, I don't even care
that much about the evals.

662
00:37:42,682 --> 00:37:46,382
I think it's more interesting to
consider that all of these models are

663
00:37:46,382 --> 00:37:51,252
going to have different tricks ranging
from those five words that Anthropic

664
00:37:51,262 --> 00:37:56,512
had to do all the way up to like Q
star with test time compute type stuff.

665
00:37:57,112 --> 00:37:58,992
The thing that bothers me is video.

666
00:37:59,052 --> 00:38:01,532
And I just thought about
what the equivalent is.

667
00:38:01,552 --> 00:38:06,392
Remember like a decade ago Apple
started marketing their new cameras

668
00:38:06,392 --> 00:38:10,482
by showing you the output of the
iPhone camera when they announced it.

669
00:38:10,512 --> 00:38:13,392
And then I don't know
whether it was Samsung or LG.

670
00:38:13,432 --> 00:38:20,492
And when they announced it, they shared
photos from DSLRs and they silently

671
00:38:20,492 --> 00:38:23,672
just wanted people to infer that
that was the image quality that was

672
00:38:23,672 --> 00:38:27,792
coming from the phone, and then people
discovered within an hour that it was a

673
00:38:27,802 --> 00:38:30,552
digital, like, SLR that took the photo.

674
00:38:30,812 --> 00:38:32,702
That feels exactly what happened here.

675
00:38:33,208 --> 00:38:37,678
And I'm sure that the demo with a little
bit of rough edges that they would have

676
00:38:37,678 --> 00:38:42,338
had if they had shown us the prompt steps
in between and the wait for an inference

677
00:38:42,338 --> 00:38:45,698
to occur still would have been a magical
moment and people would have lost their

678
00:38:45,698 --> 00:38:51,998
minds, but because we feel misled, it
erodes our trust and we feel betrayed,

679
00:38:51,998 --> 00:38:53,208
which is a very funny thing to say.

680
00:38:53,418 --> 00:38:57,023
This reminds me of A moment that
has surprised me, and there's a

681
00:38:57,023 --> 00:39:00,493
lesson here broadly for founders,
and it's not just, you know, be

682
00:39:00,493 --> 00:39:02,333
honest in your marketing material.

683
00:39:02,343 --> 00:39:07,253
I knew when I was a founder that the
common wisdom was just be completely

684
00:39:07,253 --> 00:39:11,833
upfront with VCs because they have seen
so many pitches that they can sniff out

685
00:39:11,833 --> 00:39:13,493
when something doesn't sound correct.

686
00:39:13,763 --> 00:39:17,393
I will tell you in a pitch a couple
of months ago you may not remember it.

687
00:39:17,893 --> 00:39:24,333
There was one moment where you paused, you
raised an eyebrow, you asked one question.

688
00:39:25,633 --> 00:39:30,783
And it was, it was not an aggressive
question, but it, it pulled the

689
00:39:30,783 --> 00:39:33,683
first thread that got to the truth.

690
00:39:34,773 --> 00:39:39,288
And my sense is, in that case, if
he had just been up front We reach a

691
00:39:39,298 --> 00:39:42,958
slightly different outcome versus having
to pull that thread and discover that

692
00:39:42,958 --> 00:39:47,598
there was a little bit of, of deception
in how he was presenting things.

693
00:39:47,788 --> 00:39:50,268
Nabeel Hyatt: Oh, well, to be clear, he
was trying to put a gloss on everything.

694
00:39:50,298 --> 00:39:54,378
And I do remember exactly that
meeting the company had gone through

695
00:39:54,458 --> 00:39:59,648
a pivot, some founder breakup y
stuff, you know, just a lot of change.

696
00:40:00,078 --> 00:40:01,038
And I think.

697
00:40:01,463 --> 00:40:04,443
And had been around for a little bit,
all things we didn't know when that

698
00:40:04,443 --> 00:40:12,053
founder came in to present, but we
take first meetings all week long,

699
00:40:12,263 --> 00:40:15,413
like, if we're not good at reading
people and figuring out what's really

700
00:40:15,413 --> 00:40:16,963
happened, you can't do this job.

701
00:40:17,593 --> 00:40:22,748
And meanwhile, the most important part
of this job is Establishing whether the

702
00:40:22,748 --> 00:40:26,068
other person across the hall is authentic
and you can trust them, because you're

703
00:40:26,068 --> 00:40:27,818
going to be on a long journey together.

704
00:40:28,208 --> 00:40:34,048
And so, before it's a good business
model or it's an amazing product or it's

705
00:40:34,058 --> 00:40:37,688
somebody you want to work with because
you love the intellectual banter and you

706
00:40:37,698 --> 00:40:40,168
think they're going to be a great leader
or whatever else is going to get you

707
00:40:40,168 --> 00:40:43,908
excited about this startup, you can't do
it if you don't think they're all being

708
00:40:43,908 --> 00:40:45,558
authentic and real and honest with you.

709
00:40:45,648 --> 00:40:49,583
And so, Yeah, that was a founder who
had clearly gone through some stuff,

710
00:40:49,946 --> 00:40:52,496
Fraser Kelton: we don't care if
they've gone through stuff, right?

711
00:40:52,566 --> 00:40:53,036
Of course.

712
00:40:53,346 --> 00:40:53,856
that's part of the

713
00:40:54,041 --> 00:40:57,851
Nabeel Hyatt: we would love if gone
through stuff, like, you learn some

714
00:40:57,851 --> 00:41:04,291
lessons, just own it, and tell the
story about how your, you started

715
00:41:04,311 --> 00:41:07,106
thinking it was this other thing
and you were just wrong, or, Bye.

716
00:41:07,396 --> 00:41:10,186
You moved to this town and it was
the wrong town because there was

717
00:41:10,186 --> 00:41:11,306
just a bunch of fly by nights.

718
00:41:11,306 --> 00:41:15,226
You took this founder on board, but they
were just a ne'er do well, so you had

719
00:41:15,226 --> 00:41:20,506
to get rid of them or just whatever it
happens to be that, that you went through.

720
00:41:20,516 --> 00:41:22,416
You just want your learned insights.

721
00:41:22,486 --> 00:41:26,296
And I think way too often people
want to tell a glossy story about how

722
00:41:26,296 --> 00:41:28,886
everything's up and to the right and
you got to get on board right now.

723
00:41:29,226 --> 00:41:33,156
Because this round is, oh, the other trick
is like this round is closing in two days.

724
00:41:33,206 --> 00:41:35,696
This stuff, like create senses of urgency.

725
00:41:36,096 --> 00:41:39,796
None of that stuff, all
of that stuff just hurts.

726
00:41:39,826 --> 00:41:42,926
If you have a slow fundraising
process, first of all,

727
00:41:42,926 --> 00:41:44,266
people probably already know.

728
00:41:44,276 --> 00:41:46,186
Just say, I think they're all dumb.

729
00:41:46,456 --> 00:41:49,636
This is the reason I think you are
going to be smarter than all of them.

730
00:41:49,636 --> 00:41:51,686
You can try and go to their ego.

731
00:41:51,876 --> 00:41:53,746
So I'm not saying you
don't try to storytell.

732
00:41:54,066 --> 00:41:56,576
I'm just saying you have
to know how to do it with.

733
00:41:57,096 --> 00:42:00,046
Being yourself and being authentic
to the journey that you've been on.

734
00:42:01,046 --> 00:42:01,416
Fraser Kelton: Yep.

735
00:42:01,466 --> 00:42:03,883
How many times have we seen somebody,
oh, okay, the deal's coming together

736
00:42:03,883 --> 00:42:06,506
in two days, you have to move quickly,
and then we're like, okay, well

737
00:42:06,506 --> 00:42:08,056
then this is not the deal for us.

738
00:42:08,756 --> 00:42:10,996
And then all of a sudden you see
them try to backtrack fairly quickly.

739
00:42:11,346 --> 00:42:14,366
Well listen, we really like you, so maybe
we can give you a couple extra days.

740
00:42:14,386 --> 00:42:15,176
And you're like, alright

741
00:42:15,336 --> 00:42:20,346
Nabeel Hyatt: I had the exact opposite
thing happen to me last week where I had

742
00:42:20,346 --> 00:42:26,976
a founder email in and I passed over email
and I wrote up, but I wrote like a good

743
00:42:26,976 --> 00:42:30,466
little paragraph about why, like this is
the thing that these are the reasons that

744
00:42:30,486 --> 00:42:31,366
I'm not sure you're going to be there.

745
00:42:32,416 --> 00:42:35,466
And I've gotten, mostly you get
crickets, they're going to move on,

746
00:42:35,496 --> 00:42:37,236
which is totally understandable, right?

747
00:42:37,716 --> 00:42:41,506
The second thing you get is defensive,
angry feedback that I'm just dumb.

748
00:42:41,896 --> 00:42:44,976
Which I'm not sure exactly what
that sales tactic is, but whatever.

749
00:42:45,656 --> 00:42:48,236
I got back from that
founder you might be right.

750
00:42:49,586 --> 00:42:52,176
Here are the things that I
think I've worked through.

751
00:42:52,656 --> 00:42:57,176
To try and prove what you're saying
wrong, and then gave a couple of little

752
00:42:57,176 --> 00:43:00,086
notes of the other things they tried
that don't come out, of course, in

753
00:43:00,086 --> 00:43:03,676
the one paragraph pitch of the other
versions of that business over time,

754
00:43:03,856 --> 00:43:05,506
the struggles they've had, and so forth.

755
00:43:05,906 --> 00:43:08,966
I mean, I got on a Zoom on that
person, like, three hours later.

756
00:43:10,646 --> 00:43:11,176
Fraser Kelton: Yep.

757
00:43:11,316 --> 00:43:12,116
Nabeel Hyatt: I was like, Oh, you're

758
00:43:12,206 --> 00:43:12,556
Fraser Kelton: get it.

759
00:43:12,586 --> 00:43:13,196
I get it.

760
00:43:13,506 --> 00:43:14,156
Nabeel Hyatt: this business.

761
00:43:14,176 --> 00:43:16,096
And you're authentically
trying to engage with me on it.

762
00:43:16,096 --> 00:43:17,086
And you're not combative about it.

763
00:43:17,086 --> 00:43:18,376
You're just having a
conversation about it.

764
00:43:18,376 --> 00:43:19,766
Like, awesome.

765
00:43:20,016 --> 00:43:22,796
And I, you know, didn't turn
into an investment that day.

766
00:43:22,836 --> 00:43:25,756
It may in the future we'll
see, but I certainly hold that

767
00:43:25,756 --> 00:43:26,946
founder in really high regard.

768
00:43:27,946 --> 00:43:28,356
Fraser Kelton: I get it.

769
00:43:28,406 --> 00:43:34,576
It was amazing to see, some depth
of experience such that you just,

770
00:43:35,266 --> 00:43:39,956
you knew based on two sentences
that something wasn't right.

771
00:43:39,956 --> 00:43:44,116
And it just reinforced what I had
been told when I was a founder.

772
00:43:44,641 --> 00:43:46,091
Don't bother, right?

773
00:43:46,416 --> 00:43:48,796
Nabeel Hyatt: It's similar when
you're pitching and you don't,

774
00:43:48,886 --> 00:43:51,786
you're trying to gloss over the
particular risks or problem with your

775
00:43:51,786 --> 00:43:54,386
startup, the old real estate trick.

776
00:43:54,741 --> 00:43:57,771
That Realtors use is when they
show you a house they list all

777
00:43:57,771 --> 00:44:00,081
the wonderful things and then, and
then they're doing the walkthrough.

778
00:44:00,091 --> 00:44:03,921
They talk about the one thing that's
the problem with this house and what

779
00:44:03,921 --> 00:44:06,641
they're trying to do focus your time
and energy on the one thing so you

780
00:44:06,641 --> 00:44:07,901
don't think of the 30 other things.

781
00:44:08,291 --> 00:44:10,881
That's very different from
authentically having a conversation

782
00:44:10,881 --> 00:44:12,361
with an investor about your business.

783
00:44:12,671 --> 00:44:15,191
But similarly, these are
early stage startups.

784
00:44:15,421 --> 00:44:17,831
There's no way nothing is
wrong with your business.

785
00:44:18,126 --> 00:44:18,446
Fraser Kelton: Yeah.

786
00:44:18,881 --> 00:44:21,691
Nabeel Hyatt: And so you might as
well talk about the things that you

787
00:44:21,701 --> 00:44:24,831
think are really risky or are broken
or that you haven't figured out yet.

788
00:44:25,386 --> 00:44:27,866
Because the right investor is going to
be the person that's going to be like,

789
00:44:28,156 --> 00:44:31,356
I don't think those are real risks, or
I'm willing to take on that risk, or

790
00:44:32,166 --> 00:44:35,516
like, I think you can solve that risk,
and that's, that's the right way to

791
00:44:35,516 --> 00:44:37,076
have the conversation about the path.

792
00:44:37,366 --> 00:44:40,526
Nobody expects these things to be
totally finished and that's a very,

793
00:44:40,546 --> 00:44:40,936
Fraser Kelton: sure.

794
00:44:41,036 --> 00:44:41,376
Yeah.

795
00:44:41,676 --> 00:44:44,506
Somebody internally here said
that the quickest way to a no

796
00:44:44,546 --> 00:44:47,456
is when there is no risk, right?

797
00:44:47,456 --> 00:44:49,126
Because that's not a venture business.

798
00:44:49,126 --> 00:44:50,416
That's, that's not for us.

799
00:44:50,516 --> 00:44:53,396
Nabeel Hyatt: When a founder feels like
They know the problems they think they've

800
00:44:53,426 --> 00:44:59,136
kind of solved and the areas where they're
self reflective and self aware enough to

801
00:44:59,136 --> 00:45:02,016
realize they've got a lot of work to do.

802
00:45:02,606 --> 00:45:06,826
And you can have an open and honest
conversation about doing that work..

803
00:45:07,146 --> 00:45:07,536
Fraser Kelton: Mm hmm.

804
00:45:08,206 --> 00:45:11,026
I also think the challenge here is
that every firm is different, right?

805
00:45:11,226 --> 00:45:15,026
And so, whenever anybody has
shown us a demo, you see almost

806
00:45:15,026 --> 00:45:16,556
everybody in the room lean forward.

807
00:45:16,566 --> 00:45:21,646
And when people have had stilted
presentation pitch mode, everybody's,

808
00:45:21,676 --> 00:45:23,176
you know, kind of in lean back.

809
00:45:24,051 --> 00:45:26,751
Nabeel Hyatt: At Spark, we like
demos, we like talking product, and

810
00:45:26,911 --> 00:45:29,801
that's, you know, but you're right,
that's not how it is at every shop.

811
00:45:30,151 --> 00:45:32,241
That's not that's not how
lots of investors operate.

812
00:45:33,491 --> 00:45:36,141
Fraser Kelton: that would
probably be the challenge here is

813
00:45:36,151 --> 00:45:37,741
everybody operates differently.

814
00:45:37,831 --> 00:45:42,881
That's where you have an opportunity in
these moments to find the person that

815
00:45:42,881 --> 00:45:44,491
you want to be with for a long time.

816
00:45:44,581 --> 00:45:44,841
Right?

817
00:45:44,841 --> 00:45:48,431
So there, there will, there are
different founders who appreciate

818
00:45:48,451 --> 00:45:49,671
different types of techniques too.

819
00:45:49,726 --> 00:45:52,546
Nabeel Hyatt: It can feel from the
fundraising side, and I certainly felt

820
00:45:52,546 --> 00:45:55,936
it as a founder, like, I just want
somebody to give me a first term sheet.

821
00:45:55,956 --> 00:45:58,116
I'm just trying to raise
capital, whoever it can be.

822
00:45:58,686 --> 00:46:03,236
But that's a little bit like in today's
age, like applying to college by just

823
00:46:03,236 --> 00:46:06,826
saying, I, I really love your school
because it's a great school for learning.

824
00:46:06,856 --> 00:46:09,286
And it's, that's not a great
way to get into college.

825
00:46:09,586 --> 00:46:13,436
I had an admissions person at NYU tell
me that they, the admissions people

826
00:46:13,436 --> 00:46:17,066
there, they always do a thing where they
cover up, why do you want to go to NYU?

827
00:46:17,336 --> 00:46:23,026
And if the answer is you could put in
Columbia instead of NYU the answer, then

828
00:46:23,026 --> 00:46:24,996
that's not the person for NYU, right?

829
00:46:25,436 --> 00:46:25,856
Fraser Kelton: Yep.

830
00:46:25,956 --> 00:46:26,506
Yep.

831
00:46:26,556 --> 00:46:28,726
Nabeel Hyatt: know, it's, if
it's, I love the opportunity for

832
00:46:28,726 --> 00:46:31,606
internships in the dynamic city
and, you know, stuff like that.

833
00:46:31,606 --> 00:46:32,776
It's like, that's not really about NYU.

834
00:46:32,796 --> 00:46:33,566
That's about New York.

835
00:46:33,916 --> 00:46:37,871
so I think similarly when fundraising
The thing I got to in the latter

836
00:46:38,101 --> 00:46:44,041
half of my third, fourth startup in
fundraising was, um, I'm going to pitch

837
00:46:44,101 --> 00:46:47,138
the way I want to pitch, not the way
my founder friends tell me to pitch.

838
00:46:47,138 --> 00:46:50,941
And I'm going pitch in a way that is
authentically me and the way that I

839
00:46:50,941 --> 00:46:53,821
want to talk about how I want to raise,
run this company, the culture I want

840
00:46:53,821 --> 00:46:55,481
to build, the problems my startup has.

841
00:46:55,791 --> 00:46:58,031
I'm just going to lay it
on the table authentically.

842
00:46:58,301 --> 00:47:00,431
And then the job isn't to find.

843
00:47:00,931 --> 00:47:01,851
50 term sheets.

844
00:47:02,301 --> 00:47:05,251
The job is to find one or two term sheets.

845
00:47:05,491 --> 00:47:09,431
If I, if I can pitch the way I want
to pitch my business I'll get lots

846
00:47:09,431 --> 00:47:11,651
of strong no's, but one strong yes.

847
00:47:12,166 --> 00:47:17,556
And lots of strong no's, but one or
two or three strong yes's, it is ten

848
00:47:17,556 --> 00:47:20,406
times more valuable than a bunch of meh.

849
00:47:20,886 --> 00:47:25,336
This seemed okay, because those
don't lead to board seats and checks

850
00:47:25,386 --> 00:47:28,856
and people who are going to join
your cause for the next ten years.

851
00:47:29,478 --> 00:47:33,118
Fraser Kelton: So having listened to
that and then having a moment to reflect.

852
00:47:33,168 --> 00:47:37,978
The thing that I would do differently
that I think would have a material impact.

853
00:47:38,813 --> 00:47:43,793
is to have a very authentic opening
as to why I was excited to have

854
00:47:43,793 --> 00:47:47,723
this conversation with this specific
person in this specific firm.

855
00:47:48,403 --> 00:47:51,313
You and I had the joy of sitting
with that founder a couple of months

856
00:47:51,313 --> 00:47:54,813
ago now, who said, I'm excited
at the prospect of working with

857
00:47:54,813 --> 00:47:57,983
Spark because you have a history of
supporting founders doing brave things.

858
00:47:58,013 --> 00:48:00,573
And I know that that worked.

859
00:48:00,708 --> 00:48:05,198
On you and I because we independently
said it with other people after the fact,

860
00:48:05,369 --> 00:48:05,539
Nabeel Hyatt: good

861
00:48:06,068 --> 00:48:08,708
Fraser Kelton: And it was
gr it was great sales.

862
00:48:08,738 --> 00:48:12,248
It, but it was, well, just like
any great sales, it was authentic

863
00:48:12,248 --> 00:48:13,898
and it, and it resonated, right?

864
00:48:13,984 --> 00:48:16,914
Nabeel Hyatt: and you could feel it in
the tone of when they were saying that,

865
00:48:17,224 --> 00:48:18,784
it was something they were really feeling.

866
00:48:19,034 --> 00:48:22,324
What do you do when you're
pitching, X, Y, Z, fund?

867
00:48:22,904 --> 00:48:26,744
That you don't really know why,
why you're talking to them.

868
00:48:26,754 --> 00:48:29,604
You just, you can't figure out
the most amazing, there's no

869
00:48:29,604 --> 00:48:31,804
obvious amazing reason why you're
talking to them in the first place.

870
00:48:32,679 --> 00:48:35,574
Fraser Kelton: Why are you wasting
either of your party's time?

871
00:48:36,329 --> 00:48:37,349
Is the first one.

872
00:48:37,939 --> 00:48:43,179
If you can't put in 15 minutes of thought
and research and come up with one reason,

873
00:48:44,349 --> 00:48:45,449
then why are you talking to that person?

874
00:48:45,999 --> 00:48:49,039
Nabeel Hyatt: There are people that like,
there are CEOs that like to build very

875
00:48:49,039 --> 00:48:52,819
long spreadsheets of the 40 people that
they're going to go through and talk to.

876
00:48:52,859 --> 00:48:58,919
And look, there are times where
fundraising is really CEOs who It was

877
00:48:58,919 --> 00:49:04,819
the 38th person in the Excel spreadsheet
that you got to that raised the round.

878
00:49:04,829 --> 00:49:08,849
Personally, I have had rounds where
I have had to do that in the past.

879
00:49:09,259 --> 00:49:12,109
But I think that is different
from being casual about it.

880
00:49:12,739 --> 00:49:14,409
And I think that's what talking to.

881
00:49:15,684 --> 00:49:21,414
Long term relationships, these are
big decisions for the person on the

882
00:49:21,454 --> 00:49:26,834
other side and they can feel it when
when the work hasn't been put in.

883
00:49:26,934 --> 00:49:31,834
And so, I know for a lot of founders
raising money can feel like a quote

884
00:49:31,834 --> 00:49:34,764
unquote distraction and I want to
get back to quote unquote work.

885
00:49:35,134 --> 00:49:40,394
And I've always really hated that
phrasing because You know, getting

886
00:49:40,394 --> 00:49:44,174
rid of a board member is like 10
times harder than getting divorced.

887
00:49:44,174 --> 00:49:48,174
Like, you're, you're recruiting
somebody that is going to be

888
00:49:48,414 --> 00:49:49,954
you for a really long time.

889
00:49:49,954 --> 00:49:52,494
Like, you should put the time and
effort in the same way that you would

890
00:49:52,494 --> 00:49:54,204
to recruit a CTO or anybody else.

891
00:49:55,178 --> 00:49:56,328
Fraser Kelton: Oh, for sure, right?

892
00:49:56,328 --> 00:50:01,433
It is, it's a byproduct of COVID
maybe where it became speed dating

893
00:50:01,453 --> 00:50:05,843
and the whole community went crazy,
but the idea that you would sign

894
00:50:05,843 --> 00:50:08,083
up for this level of intense.

895
00:50:09,033 --> 00:50:13,503
Camaraderie without having a an
investment seems rather silly.

896
00:50:13,733 --> 00:50:15,143
You know, I got good news for you.

897
00:50:15,233 --> 00:50:16,223
I got good news for you.

898
00:50:16,728 --> 00:50:17,218
Nabeel Hyatt: What's up?

899
00:50:18,218 --> 00:50:19,118
Fraser Kelton: I googled superpowered.

900
00:50:21,113 --> 00:50:21,623
Nabeel Hyatt: Mm hmm.

901
00:50:21,683 --> 00:50:24,023
Fraser Kelton: And I'm, I'm reading
an article and we'll come back

902
00:50:24,023 --> 00:50:27,233
to the product, but I wanna, I
want to, I wanna live with you.

903
00:50:27,643 --> 00:50:31,543
The company says they are not
shutting down the initial product,

904
00:50:32,353 --> 00:50:32,913
Nabeel Hyatt: Yes!

905
00:50:34,980 --> 00:50:35,220
Fraser Kelton: All right.

906
00:50:35,220 --> 00:50:36,210
Let's take a step back.

907
00:50:36,380 --> 00:50:39,140
Tell us about superpowered
and why you're over the moon.

908
00:50:39,145 --> 00:50:39,275
Mm,

909
00:50:40,250 --> 00:50:45,790
Nabeel Hyatt: so, this is a great
segue into product of the week I have

910
00:50:45,790 --> 00:50:49,310
been trying to record most of my life
on a daily basis more and more of my

911
00:50:49,310 --> 00:50:54,030
life, and try and summarize it and make
it searchable and so forth the super

912
00:50:54,030 --> 00:51:01,360
powered started out as kind of like
meeting bot helper company actually

913
00:51:01,370 --> 00:51:07,360
prior to GPT and then post ChatGPT,
they turn it into an AI note taker

914
00:51:07,580 --> 00:51:11,430
for your Zoom meetings or your Google
Meet meetings and so on and so forth.

915
00:51:11,720 --> 00:51:17,080
Now if that sounds like 30 other startups,
that is because there are like 30 other

916
00:51:17,080 --> 00:51:24,070
startups that are also aI note taking
startups, folks like Fireflys, and I

917
00:51:24,080 --> 00:51:26,600
think Gong does this for salespeople.

918
00:51:26,880 --> 00:51:31,520
And you could just go open the Zoom
app store and take a look through.

919
00:51:31,520 --> 00:51:36,260
And by the way, Zoom itself has
natively launched summarization

920
00:51:36,260 --> 00:51:38,780
as well to take notes while
you're inside of your meetings.

921
00:51:39,380 --> 00:51:42,620
And so the question is, why am
I excited about superpowered not

922
00:51:42,620 --> 00:51:45,350
dying when all these things exist?

923
00:51:45,360 --> 00:51:49,250
Everyone's going to launch a
version of a product, and they're

924
00:51:49,250 --> 00:51:52,640
all going to be noisy, but the real
question is, who's done it right?

925
00:51:52,680 --> 00:51:55,700
And at least in my personal view,
I've tried all of these products.

926
00:51:56,145 --> 00:52:00,365
And none of them are good enough that
I would ever use them week after week

927
00:52:00,365 --> 00:52:02,535
after week, except for SuperPowered.

928
00:52:03,435 --> 00:52:05,085
Fraser Kelton: What,
what is super powered?

929
00:52:05,865 --> 00:52:07,155
Nabeel Hyatt: You mean,
what does the product do?

930
00:52:07,625 --> 00:52:11,255
Fraser Kelton: You just said that it's
all of the small things that they've

931
00:52:11,255 --> 00:52:15,515
done right that make it stand out from a
different AI, like transcription service.

932
00:52:16,115 --> 00:52:19,805
Like, isn't it, don't you just want
it to do reliable transcription?

933
00:52:20,385 --> 00:52:20,815
Nabeel Hyatt: No.

934
00:52:21,175 --> 00:52:24,725
First of all, nobody wants to look
at the transcription of any meeting.

935
00:52:25,415 --> 00:52:29,395
thEre is no way That I want to
actually look through all of the

936
00:52:29,395 --> 00:52:34,215
ridiculous things that I talk about
every single day, word by word.

937
00:52:34,535 --> 00:52:39,645
What you really want is summarization,
and what you really want is action items.

938
00:52:40,225 --> 00:52:43,175
And the execution on that
summarization and the execution on

939
00:52:43,175 --> 00:52:45,185
those action items is what matters.

940
00:52:45,585 --> 00:52:47,875
And it just turns out that
there's actually wildly

941
00:52:47,875 --> 00:52:50,925
variant execution on that job.

942
00:52:51,295 --> 00:52:55,195
The particularly two problems that I
have With most of the other products

943
00:52:55,195 --> 00:53:01,085
that do summarization are first,
they run inside of Zoom as an app.

944
00:53:01,585 --> 00:53:03,855
And I don't want Zoom
having control over it.

945
00:53:03,955 --> 00:53:06,105
I want my desktop to have control over it.

946
00:53:06,595 --> 00:53:09,915
And so, SuperPowered is a
desktop app, not a Zoom app.

947
00:53:10,475 --> 00:53:12,595
That's the first that matters a lot.

948
00:53:12,730 --> 00:53:13,720
Fraser Kelton: that's a big difference.

949
00:53:13,895 --> 00:53:16,525
Nabeel Hyatt: and it allows
them, in particularly, to add

950
00:53:16,535 --> 00:53:18,235
new interfaces, new Chrome.

951
00:53:18,245 --> 00:53:20,825
They can iterate on it,
like, 50 times faster.

952
00:53:21,205 --> 00:53:25,385
than trying to be one button on the
toolbar on the bottom of, of Zoom.

953
00:53:25,825 --> 00:53:28,798
It also means a user doesn't have
to ask corporate overlords whether

954
00:53:28,798 --> 00:53:33,828
they will approve this, this
app to run their infrastructure.

955
00:53:34,058 --> 00:53:36,928
Which I think is a real thing
we've got to think about in AI.

956
00:53:37,028 --> 00:53:41,238
A quick aside, I was talking to my
friend who works at Amazon and he said,

957
00:53:41,548 --> 00:53:44,818
you know, we talk in these podcasts
of all these wonderful products and he

958
00:53:44,828 --> 00:53:48,088
goes, he just, he's like, just remind
you what's going on in real life.

959
00:53:48,288 --> 00:53:54,098
Every time he even goes to use ChatGPT,
Amazon internally puts up this big

960
00:53:54,158 --> 00:53:58,188
prompt that yells at him and says,
listen, just so you know, if you put

961
00:53:58,218 --> 00:54:00,878
any confidential information into this
product, we will come and kill you.

962
00:54:01,168 --> 00:54:04,838
He can't have things scraping his
email to summarize them properly.

963
00:54:05,043 --> 00:54:09,533
You can't have products that are
helping arrange meetings through AI,

964
00:54:09,543 --> 00:54:10,793
Amazon's not getting that stuff happen.

965
00:54:10,843 --> 00:54:11,763
They're on lockdown.

966
00:54:12,143 --> 00:54:12,153
Oh.

967
00:54:12,473 --> 00:54:16,133
And anyway, so, but super powered
runs on your desktop, that's the first

968
00:54:16,133 --> 00:54:19,703
thing, and so I have control over
its use not my corporate overlords.

969
00:54:20,063 --> 00:54:23,103
And then the second thing and again, it
kind of maybe goes back to this previous

970
00:54:23,103 --> 00:54:28,023
conversation about just understanding
where we are on the entropy curve, is

971
00:54:28,023 --> 00:54:29,533
that they let you edit the prompts.

972
00:54:30,613 --> 00:54:33,203
So they have, they have meeting types.

973
00:54:33,453 --> 00:54:36,443
So for instance, if I'm meeting
a new company, I have a meeting

974
00:54:36,443 --> 00:54:38,033
type called New Company.

975
00:54:38,053 --> 00:54:40,873
And then when I meet with the founders we
work with, that's called Founders, right?

976
00:54:40,913 --> 00:54:44,583
And, and the notes I to take away,
the takeaway from each of these types

977
00:54:44,583 --> 00:54:46,483
of meetings is remarkably different.

978
00:54:47,033 --> 00:54:51,663
And of course have prompts that they put
in there that are starter prompts for the

979
00:54:51,663 --> 00:54:53,153
noob who doesn't know what they're doing.

980
00:54:53,353 --> 00:54:57,358
But inevitably, I probably for
90 percent of people today, Like,

981
00:54:57,358 --> 00:54:58,458
something's wrong about that.

982
00:54:58,488 --> 00:55:01,998
It says something in there that I
want that's not right for me, and it

983
00:55:01,998 --> 00:55:06,278
lets you open up the prompt, edit the
prompt, and get what you want out of

984
00:55:06,278 --> 00:55:10,538
it, which is the difference between
something that is kind of like, meh and

985
00:55:10,538 --> 00:55:15,848
okay, and it like gave me a couple of
interesting summarization topics and

986
00:55:15,848 --> 00:55:19,878
titles, versus I feel like an active
participant in making this thing work.

987
00:55:20,388 --> 00:55:24,528
Fraser Kelton: Don't you think that
this is also maybe why they're pivoting

988
00:55:24,528 --> 00:55:28,793
away from it in the sense that This
is a really hard problem, right?

989
00:55:29,253 --> 00:55:33,243
I was just thinking that the diversity
of meetings that people have and then

990
00:55:33,253 --> 00:55:37,983
the preferences of workflows across
those different types of meetings means

991
00:55:37,983 --> 00:55:42,713
that there's like an explosion in Quote
unquote, getting this to work well.

992
00:55:42,783 --> 00:55:46,213
Nabeel Hyatt: I think there's two
points there worth touching on.

993
00:55:47,083 --> 00:55:50,573
The first of which is that,
look, of course it's a problem.

994
00:55:51,123 --> 00:55:52,303
It's a problem.

995
00:55:52,803 --> 00:55:56,893
That there are lots of different use
cases in meetings, and it's a problem that

996
00:55:56,893 --> 00:56:01,183
this is a really busy market with lots of
competition, so it's hard to stick out.

997
00:56:02,243 --> 00:56:04,743
iF a startup doesn't want to solve
problems, then what are they doing?

998
00:56:05,333 --> 00:56:12,083
Like, I, I do worry sometimes that we,
we try and avoid all of the risk in

999
00:56:12,083 --> 00:56:16,083
our startups when problems existing
out in the world is why startups have

1000
00:56:16,083 --> 00:56:17,823
a chance to exist in the first place.

1001
00:56:18,738 --> 00:56:21,558
So you have to pick your
proper problems, but,

1002
00:56:21,653 --> 00:56:22,023
Fraser Kelton: Huh.

1003
00:56:22,103 --> 00:56:25,393
Nabeel Hyatt: but yeah, let's, but let's
spend a little time figuring out if I'm

1004
00:56:25,393 --> 00:56:29,233
a startup, how to solve this problem,
because if we can solve it, then it's

1005
00:56:29,303 --> 00:56:34,503
perfectly obvious for the 18 or 20
other AI meeting note companies that

1006
00:56:34,503 --> 00:56:36,053
they have not solved this problem yet.

1007
00:56:36,243 --> 00:56:37,043
So if I have a

1008
00:56:37,733 --> 00:56:37,913
Fraser Kelton: Mm

1009
00:56:38,143 --> 00:56:40,333
Nabeel Hyatt: then suddenly I can
explain that breakthrough very

1010
00:56:40,333 --> 00:56:43,033
clearly to my customers, and I now
have an advantage in the market.

1011
00:56:43,913 --> 00:56:43,923
Fraser Kelton: Hmm.

1012
00:56:44,148 --> 00:56:44,358
Nabeel Hyatt: So,

1013
00:56:45,023 --> 00:56:45,563
Fraser Kelton: Good, good point.

1014
00:56:45,613 --> 00:56:46,123
That's fair.

1015
00:56:46,228 --> 00:56:51,428
Nabeel Hyatt: and then my second point is
somewhat related, but I think it's also

1016
00:56:51,868 --> 00:56:53,878
back to the entropy, de entropy thing.

1017
00:56:53,908 --> 00:56:57,638
If you, if you honestly think that
we are still at the point where we're

1018
00:56:57,638 --> 00:57:01,288
trying to make all these AI products
work, then just accept the idea that

1019
00:57:01,288 --> 00:57:06,408
this is AI products are early adopter
products for right now, and they will

1020
00:57:06,408 --> 00:57:10,658
not be early adopter products, only
idiots, crazy people, only crazy people

1021
00:57:10,658 --> 00:57:12,908
like you or me might try and play with.

1022
00:57:13,548 --> 00:57:17,568
An early adopter across every
single vertical and horizontal,

1023
00:57:17,568 --> 00:57:21,818
every single week, to see how all
of these tools are developing.

1024
00:57:22,238 --> 00:57:27,808
But early adopter doesn't just
mean nerd in, in It means that for

1025
00:57:27,828 --> 00:57:30,928
somebody, this problem is so acute

1026
00:57:31,893 --> 00:57:32,423
Fraser Kelton: Mm hmm.

1027
00:57:32,918 --> 00:57:36,118
Nabeel Hyatt: will be an early adopter
to try and figure out the solution.

1028
00:57:36,463 --> 00:57:42,183
And that early adopter customer will
help you find the solution with you if

1029
00:57:42,183 --> 00:57:44,183
you give them the tools to work on it.

1030
00:57:44,683 --> 00:57:50,643
And as an, as an example, you know,
Adept, which is where an investor in,

1031
00:57:50,643 --> 00:57:54,603
is an action, they create an action
transformer model, and they've released

1032
00:57:54,913 --> 00:58:00,303
a workflow tool for building your own
little webpage navigator to take actions

1033
00:58:00,303 --> 00:58:03,173
on a webpage and do little workflows.

1034
00:58:03,643 --> 00:58:07,743
Now, the model itself is not the large
model they'll be launching relatively

1035
00:58:07,743 --> 00:58:13,513
soon, so it's an earlier model, and
the workflow tool itself is, let's

1036
00:58:13,513 --> 00:58:20,663
be honest, like, kind of hard to use,
clearly an R& D product and definitely

1037
00:58:20,663 --> 00:58:23,753
not a late adopter product that I
would give my mother or father, right?

1038
00:58:24,313 --> 00:58:24,873
But

1039
00:58:25,003 --> 00:58:25,563
Fraser Kelton: mm hmm,

1040
00:58:25,593 --> 00:58:29,118
Nabeel Hyatt: for the people who,
which Those workflows are really,

1041
00:58:29,128 --> 00:58:31,838
really acute problems in their lives.

1042
00:58:32,198 --> 00:58:34,618
They are going to trudge through it.

1043
00:58:35,058 --> 00:58:41,568
And then you will learn with your customer
versus in some R& D lab somewhere where

1044
00:58:41,568 --> 00:58:45,448
your assumptions about your customer
are wrong, which is the right way to

1045
00:58:45,448 --> 00:58:46,698
build when you're early in a market.

1046
00:58:48,023 --> 00:58:54,163
Fraser Kelton: mm hmm, the challenges that
exist today are, you know, normal in terms

1047
00:58:54,163 --> 00:58:58,883
of trying to figure out how to solve a
large, meaningful problem and that it's

1048
00:58:58,883 --> 00:59:04,203
a shame, uh, if that's the reason why
a group who had a little bit of an edge

1049
00:59:04,203 --> 00:59:08,223
on it is, is likely to not be investing
too actively into trying to solve it.

1050
00:59:08,271 --> 00:59:11,071
Nabeel Hyatt: And look, we don't
know the superpowered AI founders.

1051
00:59:11,131 --> 00:59:13,891
I don't know where they are in
funding or their traction or their

1052
00:59:13,891 --> 00:59:16,971
progress or what excites them
and gets them up in the morning.

1053
00:59:17,131 --> 00:59:21,601
I'm just a consumer of the product . But
all I know is that everybody here should

1054
00:59:21,601 --> 00:59:26,031
go to SuperPowered AI and give them
money so that they stay in business,

1055
00:59:26,041 --> 00:59:27,831
so that I can keep using product.

1056
00:59:29,426 --> 00:59:31,076
Fraser Kelton: You know, I
just skimmed the article.

1057
00:59:31,166 --> 00:59:34,956
It says that it's hard to
differentiate in this type of a

1058
00:59:34,956 --> 00:59:36,766
market to have sustained growth.

1059
00:59:37,456 --> 00:59:40,066
thEy are profitable, and so
they hope to find somebody who

1060
00:59:40,066 --> 00:59:41,406
will just continue to run it.

1061
00:59:41,706 --> 00:59:47,106
But they're pivoting to become an API
provider for anybody to create a natural

1062
00:59:47,106 --> 00:59:49,506
sounding voice based AI assistant.

1063
00:59:50,113 --> 00:59:55,883
Nabeel Hyatt: That is also a busy space
of course, but, you know, it's also

1064
00:59:55,883 --> 00:59:58,333
very possible that's just a problem that
they are more excited about solving.

1065
00:59:59,183 --> 01:00:02,833
And will, they will through the
hard difficulties of that particular

1066
01:00:02,833 --> 01:00:09,033
problem with more verve and with
more passion than they have for

1067
01:00:09,113 --> 01:00:10,363
meeting notes, which is fine.

1068
01:00:11,193 --> 01:00:14,233
Fraser Kelton: you know, yeah, people
don't often talk about that, right?

1069
01:00:14,243 --> 01:00:19,463
Is you go through a pivot and you're
doing it for a lot of You know,

1070
01:00:19,463 --> 01:00:25,873
logical reasons, but you might either
pivot into or away from an idea

1071
01:00:25,873 --> 01:00:27,483
that you actually care deeply about.

1072
01:00:27,843 --> 01:00:29,183
Nabeel Hyatt: Yeah, it's not a short road.

1073
01:00:30,223 --> 01:00:32,443
Yeah let's be done for today.

1074
01:00:32,443 --> 01:00:33,493
I

1075
01:00:33,588 --> 01:00:33,858
Fraser Kelton: do it

1076
01:00:34,273 --> 01:00:35,613
Nabeel Hyatt: think we went
through some good stuff.

1077
01:00:36,453 --> 01:00:37,833
Go download SuperPowered.

1078
01:00:38,643 --> 01:00:39,463
Give it a try.

1079
01:00:39,893 --> 01:00:41,698
We'd love to hear from
you on that product.

1080
01:00:41,728 --> 01:00:44,231
I'll also add there's a couple
other products that have launched

1081
01:00:44,231 --> 01:00:46,691
that are allowing you to do live
AI drawing if you wanna try one.

1082
01:00:46,741 --> 01:00:51,571
Leonardo AI launched a pretty good
live canvas feature where you can draw

1083
01:00:51,571 --> 01:00:54,601
on one side through a prompt and it
redraws it on the right hand side.

1084
01:00:55,636 --> 01:00:58,741
If you want to give it a shot
and then we will see you all.

1085
01:01:00,221 --> 01:01:01,601
doing one next week, Fraser?

1086
01:01:02,028 --> 01:01:02,388
Fraser Kelton: I don't know.

1087
01:01:02,438 --> 01:01:02,828
Let's see.

1088
01:01:02,868 --> 01:01:03,728
We'll see in the future.

1089
01:01:03,808 --> 01:01:04,688
We, we're not sure.

1090
01:01:05,558 --> 01:01:06,558
Nabeel Hyatt: We'll see
you all in the future.

1091
01:01:08,138 --> 01:01:08,298
Bye

1092
01:01:08,608 --> 01:01:09,088
Fraser Kelton: See ya.