Artificial General Intelligence (AGI) is the whole ball game. Whoever gets it first will rule. Who’s ahead and how to bet on them; this is only part of the excellent second chat with AI wiz Gil Syswerda. Click here to listen to part one.
Highlights:
Sal Daher Introduces Second Installment of Interview – AI and the Future
Sponsors:
Purdue University entrepreneurship, and
Peter Fasse, patent attorney at Fish & Richardson
“History is full of financial firms devising algorithms that work great in the past and don't work in the future.”
“...our back testing showed on average 15% returns per year, and our live performance was exactly that, 15% per year.”
“...we needed a supercomputer to run our algorithm. We built one. In the end, we spent upwards of a million dollars on this machine, and it took $10,000 a month in electricity to run it.”
Gil Syswerda Teams Up with the Legendary Paul Tudor Jones and His Hedge Fund via Percipio
The Start of the Sub-Prime Mortgage Crisis Breaks Gil’s Algorithm and He Goes to Cash
Investors Complain When Gil’s Fund Goes to Cash in August of 2008 but by the End of The Year, They Think He’s a Genius
The Algorithm from Gil’s Hedge Fund Finds Use at a Lead Generation Company
Percipio became Jobcase which Is Heading to Unicorn Status
After the Hedge Fund Experience Gil Returns to Angel Investing
Gil “Busted” While Coding at Darwin’s Coffee House
Genetic Rule Induction Seminar at Babson – Big Turnout
Machine Learning Talent Is Had to Get Even for the Legendary Paul Tudor Jones
Gil Starts FeatureX to Look for Patterns in Satellite Imagery
Machine Alpha Looked for Trades Based on Tudor Data
Algorithms Required so Much Computing that AWS Bill Approached $2 million in One Year
Gil Syswerda’s Next Challenge Is Artificial General Intelligence
“...if you're so smart, how come you're not a billionaire? The answer to that is I like to work on hard problems, hard technical problems.”
Sal Discusses Some of the AI Companies in His Universe: PathAI, Pilot.com and VistaPath
The First Company to Build Artificial General Intelligence Will Rule
Buy Google Shares Because DeepMind Is Doing Amazing Stuff and Could Win the AGI Race – What Else to Buy
Jeff Hawkins – Thousand Brains Theory - Numenta
AI Research Is Hugely Expensive Due to the Massive Computing Requirements
ANGEL INVEST BOSTON IS SPONSORED BY:
Purdue University entrepreneurship
Peter Fasse, patent attorney at Fish & Richardson
Transcript of, “AI & the Future”
GUEST: Gil Syswerda
Sal Daher Introduces Second Installment of Interview – AI and the Future
Sal Daher: Hey, this is Sal Daher of the Angel Invest Boston Podcast. I want to welcome you to the second episode of the Gil Syswerda interviews. The first interview was focused on his education and initial startups. This one picks up where he's about to launch his investment-related startup with some Wall Street types. Then it gets into artificial intelligence and the future of artificial intelligence. This is really entertaining, a lot of fun, and it was recorded live at PodSpot the heaven for podcasting in Venture X in Marlborough. I hope you enjoy this great, great episode.
[music]
Sponsors:
Purdue University entrepreneurship, and
Peter Fasse, patent attorney at Fish & Richardson
Sal Daher: This podcast is brought to you by Purdue University entrepreneurship, and by Peter Fasse, patent attorney at Fish & Richardson.
Sal Daher: We were talking about the problem of data mining, data snooping.
Gil Syswerda: Yes, it's a problem that we had to solve. There's a whole litany of issues with using machine learning on financial data. If we have time, maybe in the future, I have a document that I've been accumulating all the different ways that you can fool yourself on financial data.
Sal Daher: Exactly.
Gil Syswerda: Yes, there's some really serious issues you have to deal with in financial data. Data snooping is a big one. You have your machine learning algorithm do snooping, people are really good snoopers as well. You have to be very careful, for instance, of looking at the data, making some hypotheses about all of history, developing an algorithm that uses all of history, but you have snooped it.
Sal Daher: Right. Exactly.
“History is full of financial firms devising algorithms that work great in the past and don't work in the future.”
Gil Syswerda: History is full of financial firms devising algorithms that work great in the past and don't work in the future.
Sal Daher: Well, my business partner and I invested in one such company. They claimed to have an algorithm and all they were doing is back testing it. It didn't work in real data.
Gil Syswerda: It's not going to.
Sal Daher: Ultimately, those guys ended up in prison. This is a guy who the father's was a professor, a dean at one point at the Sloan School. I was lucky that I only lost I think $70,000. Well, my partner lost, I don't know, something near half a million dollars, something like that.
Gil Syswerda: Yes, I get brought in by people like you and other investors, say, "Oh, I talked to this company. They got a great algorithm, dah, dah, dah, dah, dah." It doesn't take me very long to disassemble the whole thing. They have no controls in place for some of these errors.
Sal Daher: No, but what happened is that their algorithm didn't work and they started putting money with Madoff and then the whole thing blew up, [laughs] and they [unintelligible 00:02:39] about it. Indirectly, we were investing in somebody who was putting money with Madoff, not telling us about it, eventually prison and the SEC restitution checks and all that stuff. I was mostly made whole, but my partner lost a lot of money. Anyway, Gil, how did you solve the problem of data snooping?
“...our back testing showed on average 15% returns per year, and our live performance was exactly that, 15% per year.”
Gil Syswerda: It's too complicated. It's integral to our algorithm, our hammer. We had to do a lot of stuff, but we got it to go. We got it to go in spades. When we finally went live, we were trading in the futures markets, and our back testing showed on average 15% returns per year, and our live performance was exactly that, 15% per year. If you do a graph, that went from back testing to live, you cannot tell where live training started. We were completely consistent, but it took a lot to do it. Completely non-trivial effort to try to figure out how to avoid data snooping, and all the other stuff, peeking, and there's all kinds of other stuff that can happen.
I should also mention, by the way, we have Machine Insight. We created two other companies. We created Percipio Capital Management, the hedge fund. We also created Market Insight Software. With Market Insight Software, we knew that we would be creating software to do portfolio optimization, risk assessment. We needed automated trading system, and so on. We thought that we do that kind of software in a separate company besides with Percipio.
That was the initial structure. Then we banged our heads against the wall on data snooping and other stuff. We finally got a product into the marketplace called Apex and we started trading. We all put some money into it initially.
Sal Daher: We are at the Apex Center. This is in Venture X, it's located at the Apex Center.
Gil Syswerda: It was called Apex because Fred Goff, the guy who became the CEO, he knew that I liked racecar driving. He looked up some racecar driving terms and there's, when you hit that-- going around the turn, and where you intersect the inner side of that turn, it's called the apex.
Sal Daher: Apex. Yes.
Gil Syswerda: He proposed that I said, "Sure." I actually don't think it matters very much what your financial product is called.
Sal Daher: Not at all.
Gil Syswerda: Apex was fine. We're trading. It's volatile because it is the futures market, but there's a very nice upward trend, our Sharpe Ratio was okay, our returns were okay, and so we started shopping it.
Sal Daher: Sharpe ratio is a measure of return versus risk taken.
Gil Syswerda: That's right. It's a risk-adjusted ratio. Sharpe is really all you care about in the end because, if you have a high Sharpe, you can leverage to get the returns. Later on, when we're working with Tudor, all they cared about was Sharpe Ratio because that's what you had to drive towards. We were naive, so we cared about returns. We paid attention to the Sharpe because you couldn't sell a product that had too low of a Sharpe Ratio, so then we start shopping around. Initially, we had a tech demo, and virtually no live results.
People liked our tech demos, but there's a vast, vast distrust in the financial space for back testing results because, guess what? Everybody's back testing results look good. Otherwise, they wouldn't be pitching it. It's the live performance that counts.
Then we got schooled by the market in a big way. Really, with all the advisors that we had on board, this should not have caught us by surprise, but it did. That is for the kind of product that we had the market, fully automated, it's machine-driven, you could sell that cold, hard logic, machine never gets tired, there's no Monday morning blues or anything. This thing is on all the time, and that sells, but nobody knows if it's working or not.
You could do show great tech demos or whatever, and then you have six month’s worth of real live performance and it's got it bouncing around, it's gone up to the right, but then bounce away. That could just be according to random chance. How long do you have to trade live before anybody will believe you, that you actually have something that works? Well, it turns out it's three years. You need a three-year track record.
“...we needed a supercomputer to run our algorithm. We built one. In the end, we spent upwards of a million dollars on this machine, and it took $10,000 a month in electricity to run it.”
Well, we are an expensive operation. By the way, I didn't even mention the fact that we needed a supercomputer to run our algorithm. We built one. In the end, we spent upwards of a million dollars on this machine, and it took $10,000 a month in electricity to run it. Plus, we had to pay our people and so on and so on. We didn't have enough operating capital to go for three years, so we had to raise money. That was an interesting process that we went through to raise money into a hedge fund.
Sal Daher: Well, to raise money for the management company, not for the hedge fund itself, but to support the management company.
Gil Syswerda: Well, it got structured that way, kind of a complicated structure. We had Machine Insight. We had Percipio, the hedge fund, and we had Market Insight Software. Then eventually, there was a corporate structure around Percipio, as well, as you say, a management company because some of the complexities of financial responsibilities and so on. It had to be done in just the right way, so we could take on outside equity investors.
We got a pretty strong response from General Catalyst, from Charles River Ventures, they were kind of working together. We are on a trial basis for them working together on this one. I think afterwards, they decided we're never working together again.
[laughter]
Gil Syswerda: We would be happy with that, because man, they could do a good cop, bad cop on us. You talked to one, and then you talk to the other, it's just like, "Whoa, they're talking behind our backs here." We got a deal on the table, but we also got a financial firm interested in investing in us, Natixis, the French firm. In the end, we decided to go with Natixis because we thought they could help us with sales and so on. As I recall, we raised something like $7 million from them. That gave us a pretty big runway, so we pressed on.
Now, we actually got a fair number of investments from firms, but they would want to stick a toe in the water. A toe in the water for them is a million dollars. Bit by bit, the funds under management grew to about 35 million. Part of it was that we actually grew the assets and part of it was investments. Everything was going swimmingly. We just had a solid live track record. Everything just clocked just the way we expected it to.
By then, we had met Tudor and we're developing a joint long-short equity product with them, so a launch was imminent. We roll forward now.
Sal Daher: Tudor is Paul Tudor Jones and his hedge fund, which is one of the great hedge funds in the history of investing?
Gil Syswerda Teams Up with the Legendary Paul Tudor Jones and His Hedge Fund via Percipio
Gil Syswerda: Yes, Tudor Investment Corporation. They tend to be a heavy quant shop on one side and fundamental investing on another side. Paul Tudor Jones, yes, he's world-famous. He is just one of the best traders out there, period. I got to say too, we didn't work with him directly on this venture, but the next one, I worked with him quite a bit. I've never enjoyed working with anybody more than I work with Paul. He's just extremely hardworking, super smart, and he's got a team around them. Those people never quit. Talk about an intense group of people to work with, which is good if you're a multi-billion-dollar hedge fund. It was an interesting experience. We can come back to that when we will talk about FeatureX and Machine Alpha. We're in Percipio. It's now July of 2008.
Sal Daher: Ominous times.
Gil Syswerda: Yes. Everything was peaceful and going along just fine. Nobody was worried. Our fund was up 20% already on the year, so we're only halfway through. We actually made jokes and said, "We should just go to cash at this point, because we already hit our target for the year."
Sal Daher: [laughs]
Gil Syswerda: It's only going to get worse. [laughs] Of course, we had more investment in... It's impossible to do. We're trundling along and about the second week in July, some alarm bells were going off in our system. We're a fully automated trading system. Every night, a new portfolio gets constructed. Based on what we see, it gets diffed against the old one. Some decisions are made about risk, and so on. You only play some trades, so you keep changing it.
Well, we also tracked very carefully the behavior of our portfolios against history. With a massive computation it would take--basically, we had a machine doing nothing but running these simulations because we did not want to put on a portfolio that would have a negative 10% day. We want a very, very small chance of that happening.
Service to the Listener Thinking of Becoming a Day Trader
Sal Daher: Excuse me. A little bit of a listener service. If you're thinking of becoming a day trader, this is what you're competing against. Please continue. [laughs]
Gil Syswerda: We started getting some alarm bells going off on our risk management system, that the portfolio is not behaving as expected. Now, it twiddles around all the time because there's a lot of noise in the system and so on. Those errors that accumulate over time. That, okay, if it did this thing one day, but then it disappears, that's just a fluke, so day after day after day. This thing is piling up and the needle goes into the red. We do not know what's going on. We literally do not know what's going on.
Sal Daher: What were you looking at? Credit default swaps, that kind of stuff that was priced, markets are getting thin?
Gil Syswerda: No, we were just looking at what we were trading and the price and volume behavior of our portfolios versus history. We are making certain predictions about the direction it moves and so on. That's how we made money. It became apparent that our system no longer knew what it was doing. It was effectively making random trades. Now, we weren't losing money, per se, we were losing a little bit of money because there's transaction costs. If you just do random trades, you're going to lose money because you had to pay to make these trades.
Sal Daher: You were witnessing a regime change. The patterns that existed in the market before were melting down because the market was about to collapse. [chuckles]
The Start of the Sub-Prime Mortgage Crisis Breaks Gil’s Algorithm and He Goes to Cash
Gil Syswerda: Yes. We got a very early warning on that. Obviously, it was like, we're in July. Most of the world was completely oblivious that anything was wrong. We were picking it up, but we didn't know what it was. We're not financial engineers. We're machine learning people looking at data. We just knew that something was broke. The first thing is, somebody broke our system. We re-optimized our entire thing. It took about a week on the supercomputer, every aspect of it, and then back tested it over this last period, and we got the same result. Pretty sure it wasn't our system.
The next suspect is the data feeds, so the data feeds are broken. We were buying a lot of data and computing over it, but if the primary data sources were messed up, it's very hard for us to tell. It seemed unlikely. We just didn't know what's going on. Basically, it wasn't getting better. Now, we're early August, since three, four weeks have gone by, and we have a system that's just not working. We made a big decision. This was a big, big decision to go to cash, because we're a non-discretionary fund. [chuckles]
We we're making a giant discretionary move. We were afraid we're going to screw up our three-year track record. We were at the three-year mark at this point. Marketing was going great because we sent out reports to everybody we'd ever talk to, and we're getting a lot of inbound requests for it. We thought we'd be profitable by the end of [crosstalk]--
Sal Daher: You're violating your mandate because you're non-discretionary. You didn't have discretion to go to cash. You had to be invested.
Gil Syswerda: Yes, we had to violate our own processes. We went to cash. Lock, stock, and barrel. We went to cash. Not a single trade to market. We had to inform all our investors that we've gone to cash. Well, we got--
Sal Daher: Withdrawals. Everybody thinks [crosstalk]-- Gil Syswerda: No, no. We never got a single redemption request. I mean people likes us. We did get calls like-
Sal Daher: "What are you doing?"
Gil Syswerda: -"Hey guys, we're not paying you to be a bank."
[laughter]
Sal Daher: "Have you looked at the Sharpe Ratio?"
Gil Syswerda: Natixis who was kind of a silent partner. They had a couple of people on our board, but they weren't super engaged. Although, they did try to hook us up big time with Andy Lo. We can talk about that.
Sal Daher: Yes, MIT. There's a connection with Andy Lo because of Wan Li Zhu that would have been an alternative career for him was to do a PhD with, with him.
Gil Syswerda: Well, yes. It was interesting meeting with Andy. We had several meetings.
Sal Daher: Andy Lo is professor at MIT, big quant person.
Gil Syswerda: Famous guy.
Sal Daher: A famous guy.
Gil Syswerda: Very smart. We're in cash, so we keep paper trading to see--
Sal Daher: What you would have made if you were actually in the market.
[laughter]
Gil Syswerda: If we haven't gone to cash, how much money we would lost. In the meantime, we're trying to figure this whole thing out, and we couldn't figure it out. We got a lot of pressure. Somebody from the Natixis flew from France and showed up at our door essentially unannounced. Sits us down and across our own conference table and says, "You don't understand how closely we've been watching. You really like what's been happening here." He says, "We want you to get back in the market, you chicken bleep." She didn't quite put it that way, but it was pretty strongly--
Sal Daher: We run a clean podcast. Chicken poops.
[laughter]
Investors Complain When Gil’s Fund Goes to Cash in August of 2008 but by the End of The Year, They Think He’s a Genius
Gil Syswerda: Sorry about that. Chicken bleeps. They didn't actually say that. That's my wording, but it was pretty strong indication that they really didn't like that we were in cash. We told them, once we figure it out, we'll let you know. We'll start trading again. We're never went back in the markets. The September rolls around, Lehman goes bankrupt. All chaos breaks loose. So then, by the end of the year, we're still up on the year, we haven't lost that much money. Now, our investors are saying, "My God, you guys are geniuses."
[laughter] [crosstalk]
Sal Daher: No, they're saying, "Why didn’t you short the market you idiots? You should’ve shorted the market." [laughs]
Gil Syswerda: I had that thought later. We had probably all the tea leaves there to participate in the big short. The problem is we weren't financial people. We were machine learning people.
Sal Daher: You didn't have the mandate. You were long only or you're supposed to be investing?
Gil Syswerda: No, we went both long and short. We tended to run long, but every once in a while, we would go short on something. Anyway, by the end of the year, we had lost the basis for our market because we analyzed the past to predict the future. Starting in July, the past, you know became--
Sal Daher: In the investment world, that's called a regime change. Your algorithm worked in the old regime, and that whole period until April 2009, when the market started going back up again, there's no no precedent for that, except for a few times of the depression and so on. It makes sense that your algorithm would not be working there.
Gil Syswerda: We're going to talk a little bit later about Machine Alpha a different technology where we dealt with that regime change. When you go through a period like that, not everything breaks and, besides, you can always go short if you really know that something is wrong, hence not going to go up anymore, it's going to go down. The problem that we had is that our...our system, our magic hammer and all the software around it, the risk management system and so couldn't adapt, that it strayed--the markets and how they're behaving stayed outside of what the system is capable of seeing, and therefore it started making random trades.
Anyway, we were kind of done. We didn't have enough operating capital to wait for the chaos to happen and build up a whole another three-year track record. We'd put out a few feelers and there wasn't a whole lot of interest in any of that. We redeemed everybody's money. This is like happening in early December now. Then we had to figure out what to do with the company and the technology and the supercomputer and the people.
The Algorithm from Gil’s Hedge Fund Finds Use at a Lead Generation Company
Where it finally ended up, and this is mostly driven by Dave Blundin because he had a lead gen company, he wanted to flip the whole thing into his space and see if we couldn't get a return on investment by turning the company into a lead gen company.
Sal Daher: Lead generation in what space?
Gil Syswerda: Turned out to be in the educational space.
Sal Daher: Educational space.
Gil Syswerda: For your listeners, so lead gen is a weird underbelly business over the internet. In the educational space, if you're a college you need students. If you're MIT, you have no problems. You have more students than you can deal with. You have to turn them all away. There's a lot of colleges out there and a lot of colleges struggle to get students. They want leads on people who are thinking about going back to school so they can convince them to go to their college. That's a lead. Then, the question is, how do you get a lead?
The way you do it is you put up websites, some kind or another, for instance, "We'll help you find a new job." Then along the way, it says, "Have you considered maybe going back to school and getting some more skills?" As soon as you get somebody who says, "Yes, I thought about that." You get their contact information, you go sell it to colleges, for a lot of money too. We're not talking a couple of bucks.
Sal Daher: They're just creating random websites.
Gil Syswerda: That's where the skill and trickiness comes in because you can create websites all day long, but nobody goes to them. What you have to do is you have to buy keywords on Google searches, to get your little thing to show up.
Sal Daher: This is how Wayfair was built was exactly that, creating websites and buying keywords at a time when they were very cheap on the market. They ended up having thousands of different websites, things like birdhouses.com, mybirdhouse.com, that kind of stuff, for people who wanted to buy a birdhouse. Now, it's called Wayfair. They built their business on that.
Gil Syswerda: This is how you do it.
Sal Daher: Search terms.
Gil Syswerda: You could have just one site but you can't buy the keywords for going to college, because those are too expensive. Big colleges are all over those, so you have to find cheap keywords, but the cheaper they are the sparser they are. They get really weird, way off in the weeds on keywords, so you need a lot of them, potentially millions of keywords. They're all cheap and then eventually, some of them hit.
Sal Daher: The keywords would be in the domain names somehow.
Gil Syswerda: No, it could just be any part of the search, something that triggers on a pattern match. The whole long-tail keyword thing is difficult to do. It becomes a machine learning problem. It's where this all came to be. The whole thing pivoted. Percipio became Percipio Media, I think. Now, some of the people stayed, some of the people left. By the way, Jeff Palmucci stayed for a little bit because they still wanted to try the long-short equity thing that we had built with Tudor.
I'm going, "You guys are nuts." It's the same reason we're not in the marketplace now, what makes you think that the long-short equity product is going to work in the future, given the regime that we're in. They wanted to try it anyway with other people's money, of course. The thing just crashed and burned. It didn't work either. Then Jeff was out, but Fred Goff continued on.
Percipio became Jobcase which Is Heading to Unicorn Status
It went under Dave Blundin's umbrella. Some of the investors got bought out. We all had opportunities to just get cash out of our positions, some of us stayed in and so on. Then I got to hand it to Fred, he plugged on for 10 years while trying one thing and another thing and the third thing, and then he finally found a magic formula. Percipio is now Jobcase and Jobcase is heading towards being a unicorn. They become like the LinkedIn for hourly workers.
Sal Daher: Wow.
Gil Syswerda: They're now a true "help you find a job" thing. They've gone completely off the lead gen space as far as I know. It's just rocking and rolling.
Sal Daher: Far out.
Gil Syswerda: I'm glad I stayed in it because, if I treat that as an angel investment, which is kind of sort of, for my multiple turns out to be huge at this point.
Sal Daher: That's how it is. It's one investor that makes the entire portfolio. All your angel investing is just about that one investment.
After the Hedge Fund Experience Gil Returns to Angel Investing
Gil Syswerda: Anyway, so Percipio's done. Then I personally decide I'm going to try to figure out angel investing because I've always in and out, in and out with Walnut and others. I started attending Walnut meetings again. I also invited myself into Launchpad, talked to Christopher [Mirabile] there and also in Boston Harbor. I started attending and then a few others as well, but those are the three main ones and attended every meeting. I volunteered for every due diligence call, meeting, site visit, whatever. It became my full-time job. I literally had to keep rolls of quarters in my car because I'm going from startup to startup to startup. There's always parking meters.
I got a real schooling on the area of angel investors. I tried to get a sense for who's successful and who's not, and what makes them successful and watch a lot of decision-making processes over dozens and dozens of startups, plus I got to interact with a whole lot of startups. It's really quite a bit of fun. Educational experience. I decided in the end, the person I most aligned with in terms of angel investing was Michael Mark.
Sal Daher: That's a good person to be aligned with. He's very modest. He's just, "I put money in," but he's very successful.
Gil Syswerda: Then he'll also say, "Yes, you know, the pile gets bigger over time," which does not happen with most angel investors, right?
Sal Daher: No.
Gil Syswerda: Then I started getting more and more involved with startups in terms of helping them with trying to figure out their markets and help them raise money. I was an advisor to some and on the board of a couple. I just kind of got bored with the whole thing. It's not anywhere near as much fun as doing your own startup. I did that for, I don't know a couple or three years. I decide, "All right, I'm going to go back to my roots and build another hammer."
The approach I took this time was lessons learned from Precipio with the technology we built there. What I really wanted to solve was this non-stationary markets that's what we think about it, or non-stationary data feeds where the fundamentals change. They're still predictable, in some sense, but you need to know that they've changed and you need to do something else.
Particularly the way I approach it was, there's always many, many possible signals in that data. It's just that some of them are working at some times and others are working at other times. You just have to identify how that works. Brand new start, blank slate. ECS is written in C++. I decided not to use any of it. I wanted a blank slate. I also purposely decided to do it all by myself this time, because when you're doing this kind of design there's always differing opinions about what could possibly work and what not, and the right research directions to go and so on.
Gil “Busted” While Coding at Darwin’s Coffee House
I wanted to be unfettered. I don't want to entertain other people's opinions. I had a lot of experience so far. I hadn't coded anything in a while, so I decided to do it in Java. It was a big, big undertaking. Spent over a year developing this new system called GRI, Genetic Rule Induction. Funny little story. I was in Darwin’s coffee shop in Harvard Square.
Sal Daher: Which one?
Gil Syswerda: The one right out right outside Mifflin, just down the street from Mifflin.
Sal Daher: Mifflin? Okay, that's on Mount Auburn Street.
Gil Syswerda: Just outside of Harvard Square. I have my laptop, but I'm deep in the code. It's multi-threaded and multi-machine, because it was going to be a big computational lift doing all this stuff. Writing threaded code is not for the faint of heart. You can create bugs that are so hard to find. You have to be super careful how you structure all this stuff.
Sal Daher: You need a lot of coffee. [laughs]
Gil Syswerda: I also work well in coffee shops. It's just one of my spots. I don't know how I ended up there. It was this kind of far from my house, but I must have had another meeting and I just--
Sal Daher: Great coffee. It's a great coffee shop.
Gil Syswerda: It's a great coffee shop. I'm deep into it. Suddenly, I hear right next to me, somebody say, "Gil, are you writing code?" It's like my mind comes up from the depths of Mordor and I look to see who it is, and it's Ham Lord.
Sal Daher: Ham. He's been on the podcast, several times. Anyway, Ham Lord, co-founder of Launchpad.
Gil Syswerda: He was apparently in the coffee shop and he was walking by. He saw my screen, saw the Java code on there, and he sees it's me.
Sal Daher: "You're coding? At your age, with all that gray hair."
Gil Syswerda: The way he said, it was like, I was doing something illicit. It was just like, "Gil, are you--"
Sal Daher: [crosstalk] coder of--
Gil Syswerda: I go, "Yeah," like I got caught doing something. It's a funny interlude. I got a lot of stuff working on this new system. Then by happenstance, I ran into the CTO of an ad tech company in New York, because my daughter was living there, and there was a party.
Sal Daher: Jesse?
Gil Syswerda: Yes, Jesse. He was Dag Liodden of Tapad. We're at this party and next thing you know, we're having a complete little geek fest with each other, and I'm telling him what I'm doing. He says, "That actually might be useful on our business because we need to optimize these ad campaigns to improve the number of clicks we get or conversions we get." I said, "Well, it's not quite done, but let's talk and maybe you can be a first test case."
That started an interesting working relationship with Tapad, where I eventually put up servers on Amazon. They could submit datasets to what's called an S3 bucket. There would be a watcher program that pull it in, optimize their ad campaigns and send it off to them. Got some pretty eye-popping results out of some of these campaigns, like 10 times the clicks. Conversions are harder to measure, but it looks like conversion improved as well.
I thought, "Wow, okay, we have something here." Well, it turns out, it's really hard to make money in the ad tech space by improving clicks. Kind of counterintuitive. I was thinking about this quite a bit, what's the billion-dollar idea here. The idea was, you'd have to build what's called a demand-side platform, and that's a big lift because you cannot lean startup something like this. You have to build a real-time trading system and massive data handling capabilities.
You need two sales forces, one on the supply side and one on the demand side. Then the interesting part, the machine learning part is this 5% thing, this monster company. Like, "I just really don't think I want to do this." Then I thought, "All right, so that's not the right nail. How am I going to find the right nail?" Then that's what I thought, "I should ask all the angel investors that I know," so make a presentation. I didn't want doing one at a time.
Genetic Rule Induction Seminar at Babson – Big Turnout
I send out some invitations to attend a seminar-like thing at Babson College. I used the Walnut connections to get a room. Certainly, the Walnut people, some of the Launchpad people, and then people could just invite other people, so some new people showed up. I really had no idea how many people were going to show up.
Sal Daher: It was a roomful.
Gil Syswerda: It was a roomful.
Sal Daher: A large lecture-- it was a pretty good-sized lecture hall.
Gil Syswerda: I know. I figured maybe a dozen people were going to show up, so I bought coffee and doughnuts for 20 people. That wasn't anywhere near enough. Anyway, I had a lot of fun. There was a lot of good interaction. There was one guy, in particular, who's asking question after question, really good ones on the technology trying to hone in on how in our world could this possibly work? Then I had to tell him, I can't tell you anymore without NDA. That was Dan von Kohorn.
Sal Daher: [laughs] Was Dan at Walnut already?
Gil Syswerda: No, he joined later.
Sal Daher: Later, okay. We got another Walnut member that I should have on the podcast, I've been meaning to have him on the podcast.
Gil Syswerda: Yes. He'd be interesting guy. We worked together a little bit on trying to figure some stuff out, but I didn't get any great ideas out of that whole thing, but it sure was fun to do.
Sal Daher: I love it, expanded my mind. The idea of these rules that we're able to converge to a solution really fast because of this genetic, Darwinian winnowing out of the rules. I thought it was just pretty cool.
Gil Syswerda: It is cool. It is powerful. There's other machine learning techniques out there, but GRI has its place, because it can do things other machine learning systems can't do. They can duplicate a lot of stuff that other ones can't.
Sal Daher: Anyway, after that event?
Gil Syswerda: It wasn't really that long after that. I got a call from Tudor. Paul Tudor Jones wanted to talk to me. We started having a series of meetings about machine learning applied to finance, and particularly to Tudor's trading. I gave the whole team some demos and so on because he didn't realize when he first called me that I actually had brand new technology. It's a funny story, because I was at Tudor and there's a whole roomful of people, all other technical people.
I found out later they flew somebody in from Singapore to listen to the thing. The guy was asking really good questions and later in the day, I'm talking about him, asking where he's from, blah, blah, blah, and, "Singapore," and said, "Wow, it's really fortuitous that you're here." He's, "Oh, no. I flew in for this."
[laughter]
Gil Syswerda: This meeting had been going on for two hours at least, because they were asking so many questions. They were getting really down into it and then Paul walks into the room. He says, "Look, I just got see what's going on here because nobody ever keeps these people busy for two hours."
Sal Daher: This is the legendary Paul Tudor Jones.
Machine Learning Talent Is Had to Get Even for the Legendary Paul Tudor Jones
Gil Syswerda: Yes, this is the legendary Paul Tudor Jones. He sits down and starts listening to it as well. The technology, GRI, had a lot of potential applications to finance. Paul didn't necessarily want GRI. He wanted a machine learning team. Well, that's a big ask because machine learning people, the kind that he wanted, everybody wants them.
Sal Daher: [laughs] Half of them are at Google.
Gil Syswerda: Yes. Well, with the big tech firms, hot startups. These people can write their own ticket. Money is not enough. It has to be the opportunity, it has to be the team you're working with and so on. It's not a hedge fund, off in Greenwich, Connecticut, and it's not even-- Tudor wasn't even in Greenwich. You have to arrive in Greenwich and then drive through the countryside for 45 minutes till you finally get to their place. It's like in the middle of nowhere. I said to Paul, I said, "This is not exactly the hotbed of machine learning here.
Sal Daher: [laughs]
Gil Syswerda: I said, "I don't know how you're going to pull it off, so good luck." A bit of time went on and they reached out again. Another conversation this time with the entire management team of Tudor. On the call, they said, "Gil, we've been thinking about this and we want you to figure it out." He says, "We don't know the answer either." He says, "We really like you to spend some time and figure this thing out.
Paul says, "Nothing is off the table." All right. That happened one day and one morning I wake up, I was like, "Okay." I'm sitting at my kitchen counter with a cup of coffee. I'm going, "Huh, it's like a billionaire just told me that nothing is off the table."
Sal Daher: "Nothing is off the table." [laughs]
Gil Syswerda: Surely, I could figure something out. Besides, I wanted to apply GRI and I thought, in the context of Tudor, that'd be a really nice thing because I didn't really want to start another hedge fund. Already been there, done that. It's not easy. Partnering with Tudor on the trading side sounded pretty attractive. I had the hiring problem I needed to solve. What eventually came out of that was to create two companies, two new companies besides Machine Insight again.
Gil Starts FeatureX to Look for Patterns in Satellite Imagery
One company is FeatureX and what FeatureX would do is analyze satellite imagery to extract economic and financial data out of the satellite imagery. It was pretty hot space. Orbital Insight was in it run by a friend of mine. Orbital insight, by the way, was kind of named after Machine Insight. Jimmy called me at one point and said, "Gil, I'm thinking about starting a company. Do you mind if I call it Orbital Insight?
Sal Daher: [laughs] That's wonderful.
Machine Alpha Looked for Trades Based on Tudor Data
Gil Syswerda: I said, "Yes. Fine. Yes, knock yourself out." Then the second company was Machine Alpha. Machine Alpha was very specialized and is going to apply machine learning to financial data, but very specifically to Tudor's financial data problems. Working very closely with Tudor but as a separate group, because we had IP issues that we had to resolve in terms of—I can't just willy nilly give GRI to a company that could use it and so on.
Then to keep things simple, we didn't actually incorporate Machine Alpha. We made it a group inside Feature X. Feature X was a corporate entity. Inside FeatureX was Machine Alpha group. Then we had financial arrangements about who's going to pay for what and so on. Then we were off and running. We had a very, very intense two-year development cycle working with Tudor on the Machine Alpha side and the meantime, trying to hire people into the--
Sal Daher: Yes, I remember that period, I'd get messages where they, "You know anybody in this thing or does that, or whatever." [laughs]
Gil Syswerda: It's always been challenging hiring people, good technical people. This was above and beyond anything. I needed a very particular kind of person. We can talk about these kinds of researchers if you want. The kind of person that I needed was somebody who identify themselves as being a research scientist in machine learning, but who really like to build stuff. You can identify those because I didn't need pure researchers that didn't know how to build. I didn't need engineers that didn't know machine learning, I needed a mix. It's much more powerful instead of having a researcher paired with a developer to have both those people in the same head. Well, they're hard to find.
Sal Daher: I can imagine, yes. They're you.
[laughter]
Gil Syswerda: Kind of. The thing is you can't read a resume and tell that very easily. You put feelers out, and first of all, you got to get the resumes in, but then you had this huge winnowing process to try to filter out all the hangers-on from people who can actually do some real work. Well, it turned into a major operation for us, the hiring. I probably spent half my time, maybe more on hiring.
We hired consultants to help us. We put in data systems. We hired Ed Nathanson from Red Pill, who specializes in positioning companies to attract employees. He was very helpful to us. We were just this little rinky-dink company, we had to put out a huge presence in the marketplace. We started attending all the major conferences, had recruiting booths there, looked like a big company. We handed out a bazillion T-shirts.
Sal Daher: "Fake it before you make it, kiddo." This is my old business partner, Bob Smith. The saying of Bob Smith. [crosstalk]
Gil Syswerda: It wasn't even that, we just needed to have market awareness. There's this cool company out there just. I did an analysis once about, in the entire world, how many people are there that we would consider hiring for the key R&D positions? It was in the hundreds. You got to choose what used to be called NIPS, now it's NeurIPS, a big neural network conference, they would have 10,000, 20,000 people show up. Just looking at these people--
Sal Daher: Just five of them, might have been the people that were interested in.
Gil Syswerda: Good luck finding them. By the way, everybody wants to hire these people.
Sal Daher: Yes. [chuckles]
Gil Syswerda: Eventually, did figure it out. I looked at our tracking system. We processed 3,500 candidates to hire 12 people. That's a lot of work. On the other hand, we just had the best team ever. It was so fun working with this group. We made a lot of headway on GRI and applying to Tudor's data. Eventually, back testing, kind of a smaller problem. We had to inch up to this thing. Finally, we back tested on 2008. We started way in the past because we don't want to pollute the future but, eventually, we back tested across 2008, hardly a bobble.
Sal Daher: Oh, wow.
Gil Syswerda: Yes, you see all the assets we were trading, we had some nice graphics at that point, sweeping across time. Kind of about the time that we detected that something was wrong in the market in July, everything was tending to run long then because the markets were booming. Everything's kind of going up. You want to be long most things, then suddenly, you see some red points appearing. It was calling shorts, and they got redder and redder.
By the time the financial meltdown happened, it was just pegged short. We had a little dip right at the beginning of the financial crisis, and then the performance just continued up, really high Sharpe Ratios, really nice ones. You think, "How come I'm not doing that now?"
[laughter]
Sal Daher: I was going to say, yes. Today, I was reading the T. Rowe Price, one of their top fund managers is underweight equity, not hugely, a couple of percent. He just thinks that's-- Anyway, so what happened?
Gil Syswerda: Well, while we're doing all that, first of all, we were an expensive group. People were highly paid. Tudor was paying for a lot of it, paying for office space and so on. To pull this off, our compute costs were going through the roof. We're using Amazon. We didn't build our own computer at this point. We're using big multi-core machines and using a lot of them. We eventually wrote this package, internal package called Alpha Machine, which made it really easy to specify experiments for testing new datasets and so on. It took a lot of the manual labor out of it, and it really spiked up our compute cost too because-
[laughter]
Gil Syswerda: -you really put the hammer down because you really need... you can't just willy-nilly run some thing and it cost $10,000.
Sal Daher: What was your AWS monthly bill?
Algorithms Required so Much Computing that AWS Bill Approached $2 million in One Year
Gil Syswerda: Well, it kept going up. The annual rate that we are burning exceeded a million dollars, and that's headed to $2 million a year.
Sal Daher: It's a little more than the electricity on your supercomputer back in Percipio.
Gil Syswerda: We were spending so much money on Amazon that Amazon called us, and assigned us an account rep who wanted to come visit just see what we're doing.
Sal Daher: Oh, it's not a good sign. You’re spending too much.
Gil Syswerda: No, it's we're spending too much money. At the same time, Tudor were having some headwinds in their business, because every firm has its sweet spot in trading, and a lot of firms really need some volatility in the market. We've just gone through a pretty big period of low volatility.
Sal Daher: Volatility means market moving up and down.
Gil Syswerda: Moving up and down, and if it moves up and down the right ways that you can predict you can make money. I was oblivious of that, at first, that Tudor wasn't making as much money as they were. Then we're an expensive line item. Also, the satellite space was really hot at that point, too. We put our heads together, and decided that we should pause the Machine Alpha project, and put everybody on making FeatureX a success [crosstalk]. Then certain financial provisions are made for paying people salaries and so on. They're going full tilt on FeatureX. What FeatureX did is we had a platform where made it very easy to source satellite imagery, and process it. Image recognition, recognize buildings, roads, vehicles, and so on, land use. It's a whole bunch of things you can recognize, and then do change detection across that. You can use it to monitor first of all, and then use it to detect changes over time. Now you get a time series coming out about how quickly buildings are going up, or how many parking lots are in Factory Park, parking lots, and so on. There's lots of things you can do. Then for the government, we were working on a project, this is the NGA, National Geospatial-Intelligence Agency, is one of the big five Intel agencies.
They're interested in monitoring the entire planet looking for unknown unknowns. See, I was like Donald Rumsfeld. But the thing is that it's a big planet. There are trillions of square meters and it's changing all the time. People are doing stuff, but also plants are growing and dying, the clouds going by whatever, whatever, but some changes are really significant. There's something happening. They wanted a system that could figure out changes that couldn't be explained by ordinary changes. They didn't want it based on detecting roads or something like that where it was too easy. They wanted the unknowns highlighted. Well, that's an interesting machine learning problem. That's an unsupervised machine learning problem. We designed a, what was also a pretty cool system for that. We're getting funded by the NGA but, we're running out of cash. It's because we didn't have that much revenue. We had a lot of potential, and a good team and a good product, but we hadn't quite got there.
I was taken by surprise by how hard it was to raise money because I hadn't really had problems like that in the past. What I ran into, I discovered, first of all, the East Coast investors are just not interested in this. As you can probably understand. They were even specifically telling me to go to the west coast. This is not an East Coast deal. I spent a lot of time on the West Coast and, what happened is that AI had advanced in recent years, deep learning. Not only deep learning but also open-source frameworks around deep learning that made it much, much easier to build AI systems. Capable-looking demos. Soon as that kind of stuff came out, a whole bunch of people, all the MIT people that are dorms, whatever, are building really compelling looking demos. You'll get a data set and use these frameworks and build cool looking demos. Lots of them. They'll start pitching at the VCs. What the VCs were confronted with is they can't tell a good-looking demo from real technology, a real chance. They can eventually because they bring in people like me, but at first glance, it's hard for them and they're just overwhelmed by AI opportunities. Here I come, walking along. On another opportunity, we had some nice demos and so on and so on. What they really wanted to see was, "Show me the revenue," because that's the one thing they can measure. You built something. It looks awesome. Tell me the world cares. Show me your customers. It was just a little too early and we're going to run out of funding. I have options because Tudor would have backed us and so on. I thought maybe it'd be good to actually merge an Orbital Insight. I went and talked to Jimmy because I was on the West Coast all the time and that deal came together really quickly.
Jimmy and I knew each other. We'd work together i2 in the past. Jimmy [Crawford], he is the founder and CEO of Orbital Insight. We agreed to bring the companies together. Then it happened so fast that he had to make it. He went to Burning Man. The COO and I put the whole thing together. Next thing you know, we're part of Orbital Insight. Orbital Insight is a private company. It's not like a public company where we can sell the shares. Orbital Insight is a going concern. They raised a lot of money. They got good market prospects. My CTO of Feature X / Machine Alpha Matt Falk was just like a dynamite person. He's now essentially running the entire technical organization there. About two-thirds of the people in the company reporting to him. That's a big win because Matt's a powerhouse. Knock on wood. The shares will eventually will be worth something.
Sal Daher: It's a little bit like you're owning a lot of shares a very attractive private company that someday is going to—
Gil Syswerda’s Next Challenge Is Artificial General Intelligence (AGI)
Gil Syswerda: Yes, that's exactly right. It doesn't take much imagination to imagine Orbial Insight becoming a unicorn. It's not there yet. We got some work to do. I spent some time with Orbital Insight. Then I left Orbital Insight last year. February 28, it was my last day. I started rolling into thinking about some other stuff because I really want to work on, next is AGI, Artificial General Intelligence.
Sal Daher: Why don’t you pick a hard problem?
“...if you're so smart, how come you're not a billionaire? The answer to that is I like to work on hard problems, hard technical problems.”
Gil Syswerda: I gave you a list of questions you could possibly ask. One of those, if you're so smart, how come you're not a billionaire? The answer to that is I like to work on hard problems, hard technical problems. I look around and pick the hardest dang thing I can possibly work on and then make a company out of that. It's not the easiest way to make a big company.
Sal Discusses Some of the AI Companies in His Universe: PathAI, Pilot.com and VistaPath
Sal Daher: You look at companies, for example, like some of the AI companies that I'm looking at right now, I had a chance to invest in PathAI just because of logistics. I didn't write a check to them. Basically, machine learning applied to pathology. They just raised, I think, like $160 million to buy an actual pathology lab. They bought Poplar. What are they going to do is they're going to slowly have their-- This is all public stuff. I'm not an investor. They're slowly going to convert from human pathologists to their AI, as I suspect. It would have been like when you're back in the lead generation business, or some of the other marketing ad business, driving clicks on ads, that kind of stuff, you have to build a thing around it.
What they did is I guess they talked to some VCs and to say, "Let me buy Poplar, which is a lab that has a lot of samples coming through. My AI is going to learn tons from this. We're going to slowly make it cheaper and cheaper and cheaper, and Poplar is going to take over the market, because they're going to be having AI-assisted pathology, and pathologists are expensive and so forth." Another company that is doing something like that is Pilot.com. Are you familiar with those guys?
Gil Syswerda: I'm not.
Sal Daher: Waseem Daher, Jeff Arnold, and Jessica McKellar, the three of them basically what they're doing is applying artificial intelligence to accounting to bookkeeping, starting with startups. Their last raise was valued at 1.3 billion. Whatever is out there, whatever technology is out there, but they're applying it to a particular problem, which is a very knotty problem. They're basically, eventually, they're going to have the most efficient bookkeeping operation in the entire world and everybody's going to go to them because they can underprice everybody, but right now they actually have bookkeepers.
I suspect that instead of working at the really hard problems, maybe stepping back a little bit and taking some of this stuff and just finding a vertical and applying it there. One company that I invested in recently, VistaPath. What they're doing is the very, very simple machine vision stuff for intaking pathology samples, that this is the other end of the business. PathAI is in the looking for cancer cells in pathology samples.
These guys, they receive a sample from the doctor, took out a piece of somebody's prostate and it has to be entered into the pathology process. PathAI [VistaPath actually] is doing that intake into that. I wrote them a check recently. The stuff they're doing is very, very straightforward, but it solves major, major headaches for people in the industry. It's a lot like the early stuff you were doing with scheduling software. It's like they have this thing that to them seems very mundane, but the labs are like, "Yes, we really need this because it cuts down on error. We can lower the cost. It's the first step in automating, the intake of pathology samples."
Gil Syswerda: Yes, and it's a good way of doing companies. You find a market need and then you match it up with some technology that you don't have to necessarily invent. You just have to know how to apply it, and then go make a company.
Sal Daher: The argument is instead of building the hammer, look at the problem and then borrow a hammer from somebody, license the hammer.
Gil Syswerda: Yes, building the hammer and then you have--
Sal Daher: Building the hammer is a lot more fun though. See, that's the point.
Gil Syswerda: Building the hammer is fun, but it's hard work to build an effective hammer.
Sal Daher: The point is you've gotten to be a pretty wealthy guy. Having a lot of fun, building hammers.
Gil Syswerda: Having fun building-- I like building the hammers, then I love working with a team of smart, great people to innovate in some space and solve some problems. That's why I keep doing startups. It's partly the technology development and partly the team.
Sal Daher: What's the next startup that you would do?
The First Company to Build Artificial General Intelligence Will Rule
Gil Syswerda: I'm not sure. It's hard. This is going to sound a little nuts, but where I think about things is AGI, Artificial General Intelligence. An AGI, you think of AGI 1.0. Which is a machine that can either be as good as a person or can learn to be as good as a person in practically everything. Science, engineering, law, sales, writing poetry, I mean pretty much everything. There's a big question about whether that AI needs to be embodied in a robot to sense the real-time aspect of the world that’s an open question. There's a lot of smart people who think that whoever comes up with AGI 1.0 first wins everything because nobody can catch up after that. Because once you have AGI 1.0 you can make a whole bunch of copies and have it work on making AGI 2.0, quite a bit faster than people will be able to do it. Once you have AGI 2.0, you can make 3.0, 4.0, blah, blah, boom.
Sal Daher: Well, this is a little bit of what these people are doing with PathAI, with a Pilot, and so forth. They're going to have a specialized AI for bookkeeping that picks up all the mistakes that people make, understands all the transactions people were doing, and pretty soon they're going to have a very small handful of human bookkeepers servicing thousands of customers. Then they're just going to have massive revenue and they'll take over that particular spot. You don't need AGI, you can just have a specialized artificial intelligence, which is applied in one particular vertical.
Gil Syswerda: No, I understand. There's lots of opportunities there, but the end game is super-intelligence, to create some machine entity that's thousands of times, millions of times smarter than a person. Because assuming you can keep it in its box and not take over, the economic value that is almost unlimited because that machine or a set of those machines will be able to do everything you're talking about and more. It's like all acquisition of knowledge will start coming from these machines, because they'll just be so smart. They'll run the economy. They'll solve medical problems. It just basically people doing innovative things is going to probably not be a factor in the economy anymore. People can still do interesting things and artistic things and so on, but the whole economy will be driven by these machines or by the company that owns these machines.
That's why there's a big push. Google's all over this with DeepMind. The mandate of Google DeepMind is to be the first to create an AGI. They came out with a position paper just in the last few weeks with some of their lead researchers that they think they have all the pieces now to create an AGI. It's just going to be a matter of time. Nothing new needs to be invented, nothing fundamentally new. Well, that woke a lot of people up. Woke me up because—
Buy Google Shares Because DeepMind Is Doing Amazing Stuff and Could Win the AGI Race – What Else to Buy
Sal Daher: That means you should probably buy some Google shares just to have a few of them around.
Gil Syswerda: Yes, because they could become the most valuable company on the planet. Now, DeepMind, you cannot discount them because DeepMind has done some amazing things. They are probably all reckoning, the best AI group on the planet.
Sal Daher: The protein folding.
Gil Syswerda: Yes, that's one of them. The game playing that they've done and what they've been doing, reinforcement learning, and natural language understanding, there are a lot of things that they're working on. They think that the path that they've been on for quite a number of years now using deep learning in combination with reinforcement learning can result in the whole thing. That they can achieve AGI using those techniques.
Now that's not going to be a human-like intelligence. It's probably going to be something quite different. You see that in their game playing. You remember, Go, they move forward and they have something called MuZero, where that machine can learn to play Go without being told what the rules are. I mean not only is there no human knowledge, there isn't even knowledge of the game.
All the machine has shown is a picture of the board, and has to infer everything from that. That machine learns to play Go much better than humans can play, and better than all the previous ones like AlphaGo and so on that use human knowledge or use structure of the game and so on. This machine has just come up with a way of playing Go that's not human. There's a lot of human-like strategies that it exhibits, but it makes moves that humans in thousands of years of playing this game don't make and it wins.
Well, expand that out to other kinds of realms of intelligence. It's going to be a non-human AI. It's going to be thinking different thoughts and let's hope that once they create a machine that has an IQ of a million that we can actually understand what it's saying. There's some dangers involved because if you create an AI that way, you have to be really careful about what its "motivations" are. If it has motivations, which it probably will, a game plane system has motivations, it wants to win--
Sal Daher: You might start breaking all kinds of passwords and things like that. Do you think the game—
Jeff Hawkins – Thousand Brains Theory - Numenta
Gil Syswerda: Oh, that's going to happen. I just had that-- Well, you have that conversation with your friend. Yes, current encryption is going to fall and everything that's been encrypted so far since the beginning of time is going to be un-encrypted to these machines. It's like it's not even there. Quantum encryption might be a little bit different, but that's not really out there yet.
These machines will just suck up all the knowledge that's ever been out there including anything that's been stored, like every email you've ever sent, every text message, every web page that you've visited, everything that you've purchased, your bank records. Then if necessary they'll profile you. There'll be a virtual Sal on these machines, and they can probably predict an awful lot about you. Those machines will be able to convince you of anything. They're going to be so much smarter than any of us, that they'll be able to convince anybody of anything, which of course is going to make them dangerous because it depends on what their motivations are. That's one way of AI. The non-human way of creating AI. It might work, it's a big if. You can't just take it as a given. Another way, we emulate how the brain works. There is this company Numenta that's doing that.
Sal Daher: Oh, yes. Jeff Hawkins.
Gil Syswerda: Jeff Hawkins.
Sal Daher: A Thousand Brains.
Gil Syswerda: A Thousand Brains Theory.
Sal Daher: I've gotten through the book. Thanks for referring that.
Gil Syswerda: Yes, sure. You read the book and it sounds very convincing. The brain probably does work something like that. These repeating cortical columns and so on. The nice thing about that approach is that you can create a machine that just learns. It's in its nature just to learn, to derive patterns from inputs. You don't necessarily have to have the machine act and you don't have to give it motivations, like anger and fear and all that. That's the older part of the brain. Just leave that out.
Sal Daher: The lizard brain.
Gil Syswerda: He's very comfortable killing a super intelligence. You ask it some questions and then you just turn it off at the end because it's just a machine. It's not going to care because it doesn't have any cares like that. That's a different way of creating a super intelligence, but I got to tell you, Jeff Hawkins and Numenta are taking their sweet time. I've been watching that company for many, many years now and it's always promising. Before they had the hierarchical temporal memory, that sounded promising, and now they have this.
They have a long way to go before they have a working intelligence. They have another problem too and it has to do with levels of abstraction of human technology. The world is getting more and more complicated. We're building incredible AI technology, but also just the whole stack of the internet and search and everything else. It's all built on these layers of abstraction. If you do the layers of abstraction right, one layer can operate relatively independently of the others. In computers get all the way down to the silicon, doped silicon and so on, and then your logic gates and integrated circuits, and then the structure of processors and instruction predicting, multiple cores.
Then you're finally up into microcode and then you're up into operating systems, and then dah, dah, dah, and then you're into networks and transport layers, and next thing you know you're in the internet. Nobody understands all those layers. You specialize in those things. That's where the limits of human intelligence are. The creation of those layers of abstraction, then you work within them and some portion of it. We build huge stacks of these things and get amazing things done. You look at DeepMind they're absolutely totally in that stack.
They can also use GPUs graphical processing units, which are just a happy happenstance that those things are around because they were developed to play video games.
Sal Daher: Designed to be used in video games.
Gil Syswerda: Yes, you can use them for machine learning, it got a big boost in performance.
Sal Daher: Yes, learning.
Gil Syswerda: They're building specialized processors with Google to make them more efficient for machine learning, but it's still basically just doing matrix operations. That's well and good. Hopefully, you can just keep building on all those layers of abstraction and get to an AGI. Numenta has a problem that the brain doesn't work like that. They can't use GPUs to run their stuff because the brain isn't running matrix multiplications. It's doing something else.
Sal Daher: Only consuming 28 watts. Which is astonishing.
Gil Syswerda: Yes, exactly. Massive parallelism and the whole structure of computation is different. Numenta is at a huge disadvantage because they don't have the hardware to run their algorithms on that run fast enough. They've done things like use FPGAs, which are specialized chips, you'll very quickly compute certain functions and so on. Wow, that's a big disadvantage. You’re much, much better off building on existing stack of computation.
Just to finish the thought, another path to AI is to on pure principle build up layers of abstraction for intelligence. What actually is intelligence? We've tried that in the past. That's basically the history of AI. Theorem proving and object-oriented programming, objects and links and causality and so on. You're trying to build up that stack and it hasn't been successful. We are apparently not smart enough as people to build up the right layers of abstraction so that at the top of it, you have intelligence, right? We haven't successfully done that. Given all that, we are probably going to have to grow an AI like DeepMind. Train the thing from scratch.
Sal Daher: What are they doing? Are they just training it because the artificial intelligence is very brittle? It's you train it, and then it works on the data set, it's like a regime change. When you have a regime change in the financial markets, your model doesn't work anymore. If you're applying it to something else, your artificial intelligence doesn't work anymore. If you have something that's created for self-driving cars, you're not going to have something that can play Go. Are they just stacking lots and lots of specialized artificial intelligence, and bringing it together with some overarching scheme?
Gil Syswerda: They're not going to do it that way, all right? What they're going to do is they're going to-- This is a gross oversimplification, but they're going to take deep learning networks as functional units. These deep neural networks can take a situation and abstract into what's called an encoding or embedding. Yes, it's basically vectors of numbers, but they have meaning because the network was trained to give them meaning. They hold a lot of knowledge.
That's an encoding part and then the decoder can do other things with it. I think what their plan is, is they're going to have to create lots of different embeddings, which represent a huge amount of knowledge about the world, the state of the world. Then the decoding part, we can take all that and bring it together to do whatever purpose.
Sal Daher: Okay, like the Go playing AI that they had, which didn't know the rules for Go--
Gil Syswerda: Probably a better example, as you look at some of the game plane programs like StarCraft, massive multiplayer video games, where the best players in the world now are AI systems. You have to pull together an awful lot of stuff in order to win those games. There's the visuals themselves because it's all pixels initially. Then there's strategies and planning and you have to model your opponents and their strategies and deep learning systems can pull all that together and do it.
You don't necessarily have to make that explicit. This is why they're so reward focus. They have a paper out there, it's called something like, "A reward is all you need." Pick the right reward and then the system is going to figure out how to get that reward. If it has to invent an entire new field of physics in order to get that reward, it's going to do it. [chuckles] That's an overstatement but it can, along the way, build really incredible representations just so we can achieve whatever reward you put out. Given the right structure and a whole lot of computation. That's the big disadvantage of DeepMind’s approach is you have to train these things. The learning does not work the way people learn. We can see one or two examples of something--
Sal Daher: We can really pick it up quickly.
Gil Syswerda: Right and deep learning systems today, and I'm pretty sure the deep learning systems that DeepMind has, don't exhibit that capability, yet. I would not count these people out. They know, they understand that that problem is there. They're probably just not talking about it yet.
Sal Daher: If you are able to create an artificial general intelligence, you get all the chips, all the chips in the world. If you were to invest in publicly listed companies that have a shot at doing that, which would be some of those companies?
Gil Syswerda: Well, Google for sure. Microsoft, because Microsoft has aligned itself with the other big group, OpenAI. The budgets for these groups nowadays are on the order--research groups on the order of a billion dollars a year, so they're not cheap groups to run. DeepMind has a budget of about a billion, OpenAI said they just aligned with, a few months ago with Microsoft, billion-dollar deal. Plus are now starting to purchase commercial software to generate revenue. I don't think Numenta is going to be there.
[laughter]
Gil Syswerda: I don't think any of the—
Sal Daher: Numenta is private anyway. Right?
Gil Syswerda: It's private anyway. I don't think any of the university groups--
Sal Daher: How about Apple?
Gil Syswerda: I'm not sure Apple has that focus. Apple would be a contender if they really wanted to create an AGI, but I don't think they're going down that path.
Sal Daher: Is Tesla a contender at all in there?
Gil Syswerda: Yes, I think Tesla is going to be a contender. They're sneaking up on it from a different vector but their AI Day was actually pretty impressive, ignoring the little robot thing at the end. That was a marketing gimmick by the way, to hire people. I totally understand why they did it but boy, that was a head-scratcher for a lot of people. They're going down the path of building their own hardware. They're going to solve the embodied AI problem before other people because their cars are essentially mobile robots. They deal with a real-world as it's happening. Not some sort of abstract.
Sal Daher: Jeff Hawkins likes to think that that's an important part.
Gil Syswerda: I agree, for human intelligence. It may be that being embodied and dealing with the world as it changes gives you a big lift because you can use time as a supervisor. I'm looking at you and time is going on and you're not changing much. I'm jumping all over the place there, all the pixels are changing but the Sal-ness is still there. Well, that's a powerful training thing. You can use time as a supervisor to learn. In a way that doesn't require feedback other than the fact that you just keep looking. Well, the same is true for a lot of things in the world. If you're out in the world, then you can learn a lot-
Sal Daher: You can learn an awful lot about the world.
Gil Syswerda: -just by the fact that it stays consistent over time.
Sal Daher: Maybe Waymo's going to help Google as well as DeepMind.
Gil Syswerda: I don't have much hope for Waymo. They'll don’t publish much of what they're doing but I would discount.
Sal Daher: How about a Japanese company? Are any of them working on the AGI?
Gil Syswerda: If you're looking at nations, US isn't doing it as a nation-state.
Sal Daher: No, but I'm thinking publicly listed companies, the Japanese companies, are there any? Because there are quite a few of them that are in robotics.
Gil Syswerda: I don't think they're, you know, to the point, I don't know. I don't hear much about them. I track this all the time. I have lots of news alerts on research that has happened in the field and so on. I would worry about China, simply because they could have a secret project going on. China could pull that off and maybe Russia. Certainly, Putin has made noises that AI is the single most important thing that people are working on. Now, whether they're organized enough to compete against somebody like DeepMind or OpenAI or something like that. I don't know.
Sal Daher: They certainly have the mathematicians. They have a lot of smart people in Russia.
AI Research Is Hugely Expensive Due to the Massive Computing Requirements
Gil Syswerda: I've been talking lately to some Ph.D. students and some of them have actually gotten pretty discouraged about being able to do research and move the needle in AI research because it's gotten so expensive. These models that are being built, like some of these natural language models, like GPT3, just a single optimization run to get one instance of it to go can cost millions of dollars. You have to do it a bunch of times in order to figure out what your architecture should be. There are now hundreds of billions of parameters, up in the trillion parameters. These are huge networks. They take a lot, a lot of compute power.
Sal Daher: Great. All right. Well, we have for our list, Google, Microsoft, and because of OpenAI.
Gil Syswerda: If you're looking to see who's going to win the pie. As Jeff [Sam] Altman says, all future value created by mankind from now until the end of time and the light cone of the universe. This is a guy who now runs OpenAI. There aren't very many contenders right now but there's going to be a path to that. People who build hardware, specialized AI hardware, those are good investments probably.
Sal Daher: Such as who would those be?
Gil Syswerda: Well, you could look at Nvidia for one. Intel is trying to get into the game, or it's been a little slow on that. It's going to require massive compute, and not everybody has the wherewithal to build their own hardware. Google does, and they're doing it. Shockingly, Tesla has decided they need to build their own hardware, so that's their Dojo computer. That looks eyebrow-raising, like, "Wow, Tesla's actually pulling this off." OpenAI is a little unknown because you don't know what Microsoft is doing, and Microsoft isn't really a hardware company, but they're probably going to have to, if they want to get there. There's lots and lots of stuff along the way, I keep an eye on that. I have a personal portfolio, an AI portfolio.
Sal Daher: [laughs]
Gil Syswerda: I see these things, I buy some of the shares.
Sal Daher: Good.
[background music]
Sal Daher: Thanks a lot, Gil, for making a time to be with us here at PodSpot at Venture X.
Gil Syswerda: Yes, it's been a lot of fun talking to you, Sal.
[background music]
Sal Daher: Until next time. This is Angel Invest Boston, conversation with Boston's most interesting founders and angels, in this case, angel and founder Gil Syswerda.
[music]
Sal Daher: I'm glad you were able to join us. Our engineer is Raul Rosa. Our theme was composed by John McKusick. Our graphic design is by Katharine Woodman-Maynard. Our host is coached by Grace Daher.