Noah Learner: Hello. Hello. Hello, hello. How is everybody doin' today? Anybody in? Lookin' to see who's in here. Yuriy. Hey, Yuriy, I want to check your mic, are you good?
Yuriy Yarovoy: Yeah, can you hear me?
Noah Learner: Yeah, we're great.
Yuriy Yarovoy: Awesome.
Noah Learner: Cool! So, welcome, everybody. It looks like we got four people in. Let's get started. I guess we could wait a minute. Yuriy, how's your day going?
Yuriy Yarovoy: It's good. Typical day in the neighborhood. It's a little rainy here.
Noah Learner: Excellent.
Yuriy Yarovoy: It's a Friday, and we're just ready. Nice little delay there.
Noah Learner: I love it! I had the worst echo chamber as I was getting ready, and before we get started with any of the business of this I was hoping you could educate us all about the concept of the word "John", because it had me cackling yesterday as I was thinking about life in Philadelphia.
Yuriy Yarovoy: It is a word local to the wonderful city of Brotherly Love. It literally means anything, person, place, or thing. It could substitute any part of speech. You'd be like "Hey, I'm going out for a John. Hey, did you meet that John. Hey, can you pass me the John." It literally means anything, and it is wonderful. It's always great to be in Center City Philadelphia and just hearing the word just echo around the big skyscrapers. It's fantastic.
Noah Learner: Has coming from Philadelphia framed how you think or work at all?
Yuriy Yarovoy: My first boss ever, she's awesome. For Resolution Media. Basically she said if you ever move out west you'll see a very clear difference between just the way that people approach work, and if you come from the East Coast you will be a shark in a pool of minnows. Something like that, some iteration of that. But basically people here are fairly laid back. Not where I am, in Silicon Valley it's a little more active, but generally the overall attitude out west is much more laid back. Whereas, on the East Coast it's very hungry.
Noah Learner: I felt that way, believe it or not, when I moved to Colorado. I worked in a place where there is a California guy who is head of design, and I would say things like "The client specifically asked for us to not use red", and that was me being way too aggressive. First, it's clearly communing specs and what the client ...
Yuriy Yarovoy: Yeah. So I find the West Coast much more passive-aggressive. Whereas, I like being just very direct. Like "I don't have time for this shit." I run afoul of some people's sensibilities sometimes as a result of that. So I have to tone it down a bit, but definitely would prefer to be in a much more direct environment. That's just me.
Noah Learner: Sweet. Well, for everybody who's jumping in, do you want to tell us a little bit about BlueStacks? What you guys do, tell us about your market, your niche, how big you are, subscriber count, challenges, what makes you unique?
Yuriy Yarovoy: Sure. BlueStacks, we are a PC gaming platform. We run mobile games. Think of us like Steam for mobile games. We're the 2nd largest PC gaming platform in the world behind Steam. We ticked over 370 million users in January of this past year. We have users in pretty much every country in the world. Last I checked we had 52 users in North Korea. At one point we had one or two people using us in Antarctica at McMurdo Research Station. So, really global reach. Obviously it comes with a ton of challenges, because every market is very different. Especially once you get out of western markets. Yeah, it's a lot of fun.
Noah Learner: In past lives you've kind of moved from agency to in-house?
Yuriy Yarovoy: [crosstalk 00:04:49] This is actually my first true in-house role, and I've been here for four and a half years now. I worked agency side primarily in SEO in my previous life. I don't think I'm gonna go back to that life though.
Noah Learner: Yeah, and you had shared with me yesterday a couple of those reasons. Do you want to share the PG rated ones?
Yuriy Yarovoy: Yeah! Outside of very specific markets where we don't have the personnel we don't use agencies. I have an implicit bias against agencies. Actually, it's fairly explicit, I'm telling you guys about it, right? It's mostly because generally what I find, especially at the enterprise level, agencies don't have skin in the game. So they don't really care, right? So at the end of the day they hand off some deliverables, especially on the SEO side. Those deliverables get done, they don't ... Meh, they lose a client, meh. Your Sales goes out and gets another one. So how does that make you feel as the client? I don't want to be that client. I've been on both sides of the fence, I'd rather just do the hard work and build the team internally.
Noah Learner: Yeah. Tell us a little bit about what it was like to jump into your team, and how you kind of revolutionized how you do marketing. By the way, do you want to share, or do you want me to go to the slides? Or how do you want to do this?
Yuriy Yarovoy: Yeah. Are the slides relevant at this point in the conversation? We can, I just ...
Noah Learner: No! No, no, no. Just, yeah, keep going.
Yuriy Yarovoy: You're the driver, man. I'm just like "I'm the backseat driver kind of yelling in your ear".
Noah Learner: Oh yeah, okay. Well, tell us what it was like to jump in and get started, and what specific challenges you identified early that you needed to attack?
Yuriy Yarovoy: So, when I came onboard here I came on as the SEO guy. We had a senior Marketing person who's wonderful. He was here for like two years while I was here and then he moved on. At which point I moved into his role, which is Head of Marketing. But when I came onboard ... I don't know if you wanna ... Do you want to go to like Wayback Machine and go to bluestacks.com 2014?
Noah Learner: And at that point you hadn't launched BlueStacks Live, right? The product was very different at that point?
Yuriy Yarovoy: Yeah. We had some focus on gaming, but we were also looking at more enterprise deals, or "Let's run every Android application possible", and we hadn't really narrowed down to gaming. So it was very different. It was very unfocused. Yeah, go to 2014, December. It was right at the time I joined. Yeah, it was different, right? But there was a complete lack of content here. It's clearly very focused at investors, and it's like "We're a tech product company." It's not really a consumer basing brand. So I was like "Oh shit. We got work to do, people!" So I'm very lucky that I kind of was given carte blanche, and like "Do what you gotta do. Hire who you gotta hire. Get the content that you need to get. Get designers you need to get. Get the development resources you need to get. Just do it. Get it done." Over many iterations we've arrived here, what you're seeing now.
Yuriy Yarovoy: It's very game-focused. Ultimately we are ... Our technology, we don't talk about our tech all that much. We're very focused on the end-user experience, meaning if you want to play a game, you just want to play the game. You don't really give a crap how that's being done, and we want to really sit in that background, right? We just want to get you to your game faster, and give you the best experience playing whatever game it is that you can possibly have. That's where we hang our hat.
Noah Learner: Can you talk us through how you scaled as the user base grew and expanded globally how you've thought about dealing with all of the different languages in all the different countries? Do you want to get into that now?
Yuriy Yarovoy: Sure. Yeah. That was a huge challenge in and of itself. Again, we're very lucky that we're a very global company. So we have folks on our team that's ... We have people in Saint Petersburg, we have people in Beijing, we have people in just south of Porto Alegre in Brazil, right? We have people in every major market that we have users. About 30 different countries. We have people in 30 different countries. So localization was a lot easier for us because we had those people on the ground. Obviously it takes a ridiculous amount of time, and it's just recreating the processes to actually get the content into all of these languages, but essentially we have all of the content that you see in front of you, or let's say you clink on the Star Trek banner. If you go to the language selector at the top right, yeah. So this game is available in these languages. These are all translated for the SEO geeks, and I'm sure there's quite a few that are listening.
Yuriy Yarovoy: We have [inaudible 00:11:08] Lang tags on everything. The X-default's always English. We've automated that whole process when we deploy a translation. There's a wonderful plugin for WordPress called WPML that I highly recommend. It requires some customization, at least for our purposes it did. But it works wonders. It makes things a lot easier. Then we use Trans effects, or another translation management system. There's many. We use Trans effects to actually manage the string translation that get fed into and out of WPML, which is great.
Noah Learner: Can you talk us through ... Yesterday you kind of blew my mind when we were talking about the challenges of rolling out campaigns for all the different markets, and what it takes to manage something at scale like that; and that you needed to have machines do a lot of the heavy lifting; and you couldn't rely on people. Just because of the scope of where you guys were as a company, and ...?
Yuriy Yarovoy: Yeah, yeah. You just reminded me I forgot to make that slide. Yeah. So we run quite a bit of paid media both on the display side and on the search side. Our budgets are high six figure, low seven figure per month. Our ROAS is 300% plus, as an aggregate. That constitutes maybe 25% of our traffic still at that level of spend, but when you think about games and markets that we're available in, which is global, languages that that covers, we have in just our Google Ads account ... It's really hard for me, as a side note, it's really hard for me to not call it backwards still, but that's besides the point. I'm curious how long that's gonna take. Anyway, in our Google Ads account we have hundreds of thousands of ad groups. Our MCC is ridiculous. We have at least 100K campaigns, let's put it that way; across multiple accounts.
Yuriy Yarovoy: Campaigns targeting either at a language level, some campaigns are specific to a game and have multiple ad groups. A lot of the ad groups are SKAGs, right? So we start with SKAGs and then we built them out. For those that don't know what a SKAG is, it's a Single Keyword Ad Group. So it's an entire ad group targeting a single keyword. So I can send this to Razorfish, my old alma mater, and I don't think they have the staffing capable of running all of these manually, right? So let alone for me to hire people to do this, but that's not gonna happen. It's not economical. We're a 200 person company. That would be another 50 people that are just doing ads, which is crazy 'cause it's not a huge part of our traffic to begin with. But it's all part of my mass scheme for global domination. So I wanna own every piece of the SERP, right? So we need ads. Double list on all the things.
Yuriy Yarovoy: Anyway, what we've done is we've automated the entire ads process, and this isn't just for Google Ads, we do this for Bing, we do this for Yahoo! Japan, we do this for Naver, we do this for Baidu, we do this for Yandex in Russia. Every major search engine in the world we have covered. In some respects I'm very, very lucky in the sense that we're a engineering driven company, and I work with some of the brightest minds in terms of engineering, programming code. So we spun out an entire team that's just focused on this ads product, because that's what it is, it's a product, right?
Noah Learner: Does this represent your stack? This slide that I've got going?
Yuriy Yarovoy: Essentially, yeah. So that marketing automation system is home-brewed, it's basically the brain of the operation. Our ad system does everything from ... Let's say we have a game. A game pops into our product. We automatically generate a landing page that's no-indexed, it's specific just for a campaign, right? It's totally scraped, automated. We scrape Google Play essentially, create a landing page. It's not indexed, we don't want it indexed, I don't want a huge duplicate content penalty. But it converts really, really well. We have a template that works well. We then generate based on some small set of keywords, or base keywords, like "Game name", or something. We generate either a SKAG or something more built out depending on our needs. We have many different templates. So we build out ads. I can't create a campaign if one doesn't exist, create ad groups if one doesn't exist, create ads if they don't exist.
Yuriy Yarovoy: Again, we have many, many different templates. Then we have a quite advanced bit optimization system that's all, again, built on a heuristic that we've developed based on performance, right? So every conversion, all the dollars earned, are fed back into our ad networks, or at least ad networks that allow offline conversions. There's a feedback loop and everything is optimized, and all of this is done hands-free. Obviously there's places were we can intervene. We have people that monitoring this, but at the scale that we run it's impossible for humans to control everything and be able to do it at any reasonable performance.
Noah Learner: Could you plugin off the shelf tools to replace your homebrewed stack? If so, what do you think you'd lean on?
Yuriy Yarovoy: There's this awesome tool for ads called Optimizer, which allows fairly easy scripting. So if you're not adept at Google Scripts ... Sorry, at Google Ads Scripts, you can do that, but there's really nothing off the shelf that can plugin into all of the networks that we run on. There's not. There's so many other APIs that just aren't supported, and most products are generally looking at the top of the market; and Google is so dominant that they might plugin Bing, great, but if I ask like "Hey, do you guys support Naver? Here's there API documentation." They're like "What the hell is Naver?" But it's the largest thing in Korea, and Korea's a massive market. If you're looking for hardcore gamers and you're not in Korea, you're leaving a stupid amount of money on the table.
Noah Learner: Wow. So, tell us about your target. Who's the target audience? 'Cause I'm sure I have preconceived notions that are way off mark.
Yuriy Yarovoy: What are your notions? Ha! Tables have turned! Who's the interviewer?
Noah Learner: "So there I was under the bright lights." I guess I would go with 16 to 45 year old guy.
Yuriy Yarovoy: Yeah, totally.
Noah Learner: That would be my guess.
Yuriy Yarovoy: In Japan it actually skews a little higher, up to like 55, because that culture is just very tech savvy, and they're very hardcore gamers. Korea also skews a little older, but yeah, that's perfect. Obviously mobile gamers, right? But also PC gamers that are looking to expand their library of games, because most mobile games are free-to-play. They monetize via in-app purchases, or the equivalent of downloadable content if you play console games or PC games. But they're all Freemium, essentially. So, there are a lot of games out there that you don't have to pay-to-win. They're actually a lot of fun, and very, very ...
Yuriy Yarovoy: Especially recently in the last three years you see games like Lineage 2 Revolutions, or even PUBG Mobile. Right now you can play PUBG Mobile on BlueStacks for free, and it's the full experience, granted the graphics are obviously for mobile, but they're really, really good 'cause they're for flagship mobile devices as of now. So they're essentially PCs three years ago. And you can play that for free versus going out and going on Steam or whatever and buying PUBG for 30 bucks.
Noah Learner: If you had to take yourself out of your role and you had a similar SASS challenge, and you stepped into a different role where they didn't have your system rebuilt; and you knew that you had to get to the same kind of scale, how long would it take to engineer a new solution with a whole new team and ... It just seems like a fascinating challenge that I've never really thought of.
Yuriy Yarovoy: I understand your question, and it's impossible for me to answer because it all depends on what the product is, right? For us it's a consumer product, one. So versus like B-to-B, or to sales cycle. So longer the market might be a little more narrow, right? And for us, there's an infinite amount of content, right? Because think of it ... We're not necessarily pushing BlueStacks. We're pushing this idea that you can play Clash of Clans on your PC, right? So for us the challenge is more like ... When I talk to my team I try to reinforce this point. There's two types of people in the world when it comes to BlueStacks. There's the people that have done the math, so to speak, that are like "Huh. Okay. So I have this game on my phone. I wonder if I can play it on PC." And right there we will own that person, because we're pretty much dominant in every channel that person can reach out to figure out if this solution exists.
Yuriy Yarovoy: But then there's everyone else, and that's like 80% of the market. Which is, they haven't done the math. They're like "Mobile games. PC games." And there's a huge wall between those two. Those ideas haven't combined. So my challenge is how do I introduce those people to this idea that like "Hey, you have this on your phone, you have a computer at home. That computer has significantly better specs, even if it's older, than your phone will ever have. Your battery won't die. Your connection is stable. You're not gonna eat through your data plan. Your screen is bigger. Your controls are better. You should just play on your computer. Oh, yeah, you can also use a game pad. Your entire experience can be enhanced for the game that you're clearly in love with on your phone. When you want to play on the go, you're on the train, you can play on your phone, but when you're in the comfort of your own home there's BlueStacks." So that's the idea that I really, really try to reinforce and introduce, and that's the biggest challenge. The people that have done the math, that's easy.
Noah Learner: Okay. I'm at the stage in my agency where a lot of times the types of automation solutions that I'm providing, I think a hardcore engineering team would describe as brittle. Meaning that they work great for smaller needs and smaller amounts of data pushing through Webhooks. You had a good slide that you built out yesterday. When do you get started with automation, and at what scale do you have to do it and how much of it, I guess, revolved around pain?
Yuriy Yarovoy: Yeah. No, that's a great question, and that's kind of the origin of all of our projects, right? So we basically ... I don't know if this is necessarily the right answer, this was the right answer for us, right? Let me preface this by this is by no means gospel. For instance, for ads, we looked at the scale that needed to get to. Just if you do the quick back of the napkin math there's like two million games on the Android store. Let's say 10% of those games are worth playing, right? Okay, cool. Now how many languages do we actually need to have pages in, ad copy in, all of these things? Oh, wait. Each one of those campaigns is gonna have very different bit types. Some campaign types might need CPA, a CPA bidding strategy. Some might run on CBCs, some might run on ROAS, and some might perform better on some combination of that, right?
Yuriy Yarovoy: I'm not even talking about display yet. So once you start adding that up, the scale is ridiculous. That was our problem. It was basically like "Hey, okay. We've defined the problem, our problem is we cannot possibly run this at any given scale. So what do we do?" We started running as much as we humanly could, and identified what the process for running each of these campaigns was; and to build out these campaigns. We documented all of that. Then we eventually reached the human limit of doing all of this. Then we reviewed what those processes were, right? So, how do you create landing page? What is a landing page? Are there ways that we can templatize landing pages, copy? That's partly why we run SKAGs, because you just start with a game name. Then you can look at the search terms report and see what's performing well, and you can negative a bunch of keywords. You can add a bunch of keywords at some match type.
Yuriy Yarovoy: Then you can optimize based at the keyword level. All of our rank trackings actually fed through the search terms report. So we compare and contrast like "Hey, okay. So we're ranking really well for this term. Organically we're number one. Is it worth double listing, or should we reallocate that budget elsewhere where it's more necessary?" There's necessity for it. We documented all of these processes at the limit, basically. We got to the point where things were falling apart, and here are the things that are failing first. Those are the things that we actually refined the process on and automate it first. You can go to the next slide.
Noah Learner: So before we get to the next slide, take yourself out of your role. Is there any kind of golden rule for what that human limit looks like, where it is, and how you know you're there?
Yuriy Yarovoy: If you need to scale, but there's not enough hours in the day, you're at your limit. If you are losing sleep, or the quality of your life is declining because you have more work than time, you gotta fix it, right? So you start by making your processes more efficient, because there's always room to improve those, but if you've already iterated and your process is pretty much it's leanest possible, either all of the things that can have some kind of minor automations, or ... Here's a good example. One way that we discover new games that are emerging, 'cause again, millions of apps in the Google Play store, right? Millions of games. If a game becomes popular, if a game launches ... And we have relationships with like a 150 out of the top 200 game developers. But generally they're games from small Indy developers that blow up all the time.
Yuriy Yarovoy: How do you discover those? You can't possibly have enough people on the ground reading all of the blogs in all of the communities to understand whether that's going to happen or not. That's not a scalable thing to do. We've actually short-circuited that by ... If you're on our site, we have a search bar. If a search for any game gets more than X amount of searches we send an email. We automatically create a results page that's not indexed, and then once that page gets a certain amount of ... So click on the 2nd ... like Dance From, that's not a game, but we definitely don't have a page for it, right? This is all scraped, but these pages convert. Also this isn't indexed, but basically we create this page, if this page gets a certain amount of conversions, we automatically send an email alert to our web marketing manager, and that person creates an actual page that's indexed, right?
Yuriy Yarovoy: So no one's sifting through our Google Analytics site search results. That would be crazy. It would take too much time. That would need to be done daily, right? That's a waste of people's time. So we just set up triggers for that.
Noah Learner: That's amazing.
Yuriy Yarovoy: We can email once it hits critical mass, and then we actually spend the time to create the page, right? So we've eliminated a huge part of the process by just setting up an email alert.
Noah Learner: That's awesome. That's amazing. How long does this process, how long did this take in [crosstalk 00:30:29]
Yuriy Yarovoy: For ads? Six months.
Noah Learner: Six months? Okay.
Yuriy Yarovoy: For us. Again, this is what worked for us, and we fell on our faces a bunch. I'm sure if I had to do it again it would take less time, for sure, because I kind of know where we went wrong, or where we made the wrong turn now, but there was a lot of falling down on the way.
Noah Learner: Want to go through this?
Yuriy Yarovoy: Sure! Yeah, that's kind of the same thing. You have to start with a very clear goal in mind. If you say that "I want to automate all of my processes so I never have to think ever again!", you're screwed. Just stop. That's never gonna happen. You may as well just go get cryogenically frozen, wake up in 300 years and hope that there's an A.I. around that can do it. But it ain't happening now. It's just not. So you have to have a clear goal in mind. You have to break things down to first principles. Just you have to simplify every piece of the process down to something that can be automated. Then I would really, really encourage everyone to ... I know it's not possible for everyone, but have some technical staff, or work with people that you're actually close to that you're not contracting out to, but can sit next to you and you can work through the workflow that you need automated. You start with the smallest pieces first, then you radiate outwards.
Yuriy Yarovoy: So you start with, let's say those email alerts. "Okay, cool, so now we have those landing pages. Okay, great. We have a system to generate landing pages now. They're not indexed, but they're specific for a page search, great. Okay. So next thing we need to do is, okay, we need to create campaigns. There's an EPI from AdWords, let's start there. Okay, we create campaigns. Then what's the process for defining the targeting? What's the process for creating ad groups and ads? How do you add keyword? What are the keywords that you add?" So you create these little templated steps for each one of those processes, then it's a lot easier to then plug those steps into whatever automation system that you're gonna use or build. Again, we're in kind of a different boat where a lot of our stuff's built in-house because we have the technical staff, and a lot of the functionality that we need. It's just not available out of the box.
Yuriy Yarovoy: So I'm very lucky in that regard. But, at the same time, no, you were telling me about using Zapier yesterday, and Zapier's fantastic for connecting a bunch of different data sources and passing that data along to whatever endpoint you need. There's off-the-shelf tools for 99% of things out there. It's just once you get to a point where you need to do it at a scale like, for instance, ours, or you get to a point where it's just some functionality's not available, you absolutely need some engineer sitting with you.
Yuriy Yarovoy: Man, I am the last person to talk about tech. I know enough to be very dangerous. Engineers hate when I talk to them about technology, 'cause I'm like "Oh, why don't you do it this way?" They're like "You're an idiot." So I'm just gonna preface this. That said, all of our automation backend is all Python.
Noah Learner: Python.
Noah Learner: So you connect mainly with Python?
Yuriy Yarovoy: Yeah, but that's also indicative of our backend is essentially all Google Cloud.
Noah Learner: Sure.
Yuriy Yarovoy: We run on that and AWS, and Python just happens to work really, really well on all these.
Noah Learner: It's seems like every API I look at has Python.
Yuriy Yarovoy: Yeah, it's fairly robust. My one Udacity course in Python is not going to make me an expert, no matter how much I wish it did. So I can't really say like "Oh, you can do all of these things with it", but yeah, basically from working and talking to our engineers over and over again, it seems like it's the most robust language for this sort of application.
Noah Learner: What's the Python learning tool where you get the turtle to move around? I got my kids to do that, and they were like "Oh my god! I wanna be a programmer, this is so sick! I can make a turtle move around the screen!"
Yuriy Yarovoy: That's awesome. I don't know. I did the course for Udacity, and then I also did the Codecademy one, I did two.
Noah Learner: Cool. Tell us about falling on your face in the middle as you started to get all of these little micro steps mapped out and built [crosstalk 00:35:58]
Yuriy Yarovoy: Jeff Bezos had this wonderful letter to his shareholders. I forget when. But he writes about process, and goals and process, right? He talks about when people get so engrossed in the process that the lost sight of the goals, and the process itself becomes the thing, that's a problem. For us, that happened very early on. We were like "Okay, let's create this system, and it does all of these wonderful things", but then we started focusing very deeply on each individual piece, rather than looking at the overall functionality and whether it was achieving the things that we needed to achieve. So, looking at the big picture's very, very important, and it's something that we failed ... Oh, man. It's something that we failed to do early on. Just because we were really excited, and then we didn't really know the full scope of what this was going to look like, 'cause this was kind of very opened ended at the beginning.
Yuriy Yarovoy: So we had to narrow things down, and we had to look at "Is everything that we're doing achieving the end goal?" For us, end goal was "Can we drive X amount of traffic, or can we spend Y amount of budget, or can we drive Z amount of conversions?"
Noah Learner: Do you guys use automation at all in your testing, and if so can you share what that kit looks like, or the process?
Yuriy Yarovoy: Excuse me. Oh, man. Hold on one second, I'm just gonna grab some water.
Noah Learner: It's Spring, man. If anybody's got questions, unfortunately I can't see the YouTube Live questions. So if you're in Zoom and you have questions throw them in the chat and we're gonna-
Yuriy Yarovoy: Hey, do you want to send me the YouTube Live link, and I can look at the questions?
Noah Learner: Oh, yeah.
Yuriy Yarovoy: Yeah. Teamwork!
Noah Learner: No, I just get a horrible feedback loop.
Yuriy Yarovoy: Yeah, no worries, I don't want you to open it. You wanna just ping me the link on Twitter or whatever?
Noah Learner: I'm trying to here. Give me a second. Share.
Yuriy Yarovoy: Can you do the little old school from the 60s when the TV was out of commission, like bars, like "Experiencing technical difficulties"?
Noah Learner: Well, I'm trying. I'm trying, and I don't think we have comments there. I don't think ... Yeah, there's no comments.
Yuriy Yarovoy: Okay. All right.
Noah Learner: We've got a massive audience, dude, it's huge! We're killing it!
Yuriy Yarovoy: Huge!
Noah Learner: I know. But let's see. So testing toolkit, do you guys use automation in your testing at all to drive performance?
Yuriy Yarovoy: I can't speak from the engineering end. We have a QA team that'll run through all the code that's written. I'm trying to think of what we automate ... from the website side. There's not really anything that we've automated in terms of testing for the websites beyond ... We'll take a daily crawl of the website and we'll compare DIFS, also everything's in GitHub anyways. So we can look at things that broke, but that's definitely ... Yeah, that's a good question. I can't speak from the engineering perspective, but for us right now, from the marketing side, a lot of the experimentation's manual. So the way that we kind of function, especially for [inaudible 00:39:50] keep talking about ads. But my human teammates, the humans on the team, their main job, 80% of their work, I want them to focus strictly on expanding into new markets looking for new things, experimenting.
Yuriy Yarovoy: Anything that is repetitive or focused on, or are repeatable at this point, get automated. So they'll discover something that works, document that process, pass it to our engineers, and they never do that again. That thing is automated. If it's automatable. Obviously things like, if we're running display ads, it's very difficult to generate those automatically, 'cause creatives are ... It's a little more complex than let's say text ads. But they're basically the tip of the spear, and everything that can be replicated that they do gets passed to automation and they move on and find new cool things to do. But that's the cycle.
Noah Learner: We got a great question from Nico. I'm assuming this is my friend Nico from Denver, one of the principals at Two Octobers. He says "Curious how big your dev team is, and if you outsource anything?"
Yuriy Yarovoy: No. Everything's in-house.
Noah Learner: Everything's in-house. Okay, cool. Everything. All dev work happens in-house?
Yuriy Yarovoy: Yeah, everything.
Noah Learner: Awesome. Nico hosts my favorite meetup, which is in Denver. We have the Denver Analytics meetup, and it's every 2nd Tuesday of the month? Is that right, Nico?
Yuriy Yarovoy: Nice! If I'm ever in Denver I'll definitely look you guys up.
Noah Learner: Oh, it's super fun. Yup. He said "Yup, 2nd Tuesday." Okay.
Yuriy Yarovoy: Oh, size of the dev team? It's like between 10 and 15 dedicated to just marketing products. So products servicing the marketing team so they can service the company.
Noah Learner: You have 10 to 15 devs just to support the marketing efforts?
Yuriy Yarovoy: Including QA it's on the 15 side.
Noah Learner: That is awesome.
Yuriy Yarovoy: Again, but this is atypical. I wanna stress that I have friends that also work in-house, and they're like "Holy shit. You have more resources than I have, and my company makes several zeros more than you guys make a year." It all really depends on your organizational structure, and the priority that's what you're doing kind of drives. One thing that I will tell you, and the reaching the human limit is the only way that we allocate resources, dev resources, right? So basically every new marketing product that has every rolled out, whether it's the Affiliate program, or whether it's ads automation, or any of the stuff, we have to do manually until we could not do it anymore, and that was the MVP, the Minimal Viable Product, the human way. Only then when we proved that it works did we ever move to the automation, or brought engineers back into it. Engineers won't even listen to us until we have an MVP.
Yuriy Yarovoy: I think that's the way it should be. If you start with the engineers you're gonna get scope creep, essentially, right? You're gonna be like "Oh cool. I have someone that can do this. Hey, how about this cool thing? And this cool thing?" That's a recipe for failure. So, reach your human limit, have that MVP in place, or at least the process for that in place. Prove that it works, and then automate. Don't automate first.
Noah Learner: I wanna make sure that we have time for the big surprise at the end. How much time do you need for that?
Yuriy Yarovoy: It's not really ... Yeah, it's a great surprise.
Noah Learner: Two to five minutes?
Yuriy Yarovoy: Yeah, at most.
Noah Learner: Okay, cool. I'm just watching the clock. I know everybody's got stuff to do. Let's see. Can you tell us about experimentation? Do you use automation in your experimentation? Both on the SEO and TPC side? Not from the technical side so much, but as the designing, the concept, or the scope for it, can you give us insight into that?
Yuriy Yarovoy: Can you go into more detail what you mean by designing the concept and scope?
Noah Learner: Okay. Let's say you want to know about great landing page design. You've done a bunch of testing, you've picked a winner, you've run with it for a while, and then you're seeing metrics that aren't quite what you like, how do you use automation to execute it if at all? And I'll have more questions after that.
Yuriy Yarovoy: Sure. For things like Conversion Rate Optimization, let's say your example of ... We have a page template, let's say, we use to optimize your BWO, or Google Optimize, or some CRO style tool to split test, Multivariate test, whatever. Some design, and we're like "Hey, if this design 1.4 works better than every other design, great. Let's go with that." If we want to test further, that we don't automate those tests. Those we run very manually, because ... it's very easy to lean on tools in places where I think you need pure human input. I've run into issues with tools like Optimizely, or Google Optimize, where "Oh, we have 95% confidence in ... This page is the best page and then it performs significantly worse over the long-term", right? It's because of some very simple UX issue that we didn't see, we just trusted the numbers.
Yuriy Yarovoy: Then once it lives out in the wild for weeks, you get to see that problem. So I think when you combine both the human input and that tool set, you get a much better result than just relying on the tool set.
Noah Learner: There was something that we talked about yesterday, that I'm gonna shift gears a little bit, that I thought was pretty interesting. I went through a lot of the content on your site, and I got into your Streamer Academy and I thought that that was pretty cool. We talked a little about how your affiliate section of your site works, and how you use that to amplify your messaging; and how you have been able to automate the whole system. Can you talk a little bit about that, or is that secret sauce?
Yuriy Yarovoy: No. Basically what we realized was, at least to build a brand and reach that wider audience, remember the people that did the math and the people that didn't, to reach the people that haven't done the math, so to speak, leaning on other people relevant in the gaming space and using their megaphone to push our message is significantly better. I know Rand Fishkin went off on a long discussion about the viability of influencer marketing in the long-term. I can tell you without a shadow of a doubt in the gaming space influencers work 100% of the time. Now, if you're a PC platform and you have some Kim Kardashian style influencer on Instagram, you're not going to go far with that.
Yuriy Yarovoy: It's gonna be very difficult to measure. But if you're a PC platform and you're on YouTube, or you're on Twitch, or you're on Dlive, or YouTube Gaming, or any of these platforms, it could be one, incredibly powerful for your brand to just gain exposure. Two, as long as you have the ability to, you can track everything end-to-end from the person that watches the video to the moment they install your product.
Noah Learner: I knew that it worked when my kids ... My kids are 10 and 12, and when they came to us in December and said that they wanted a Pewdiepie, they didn't say "I want a sweatshirt", "I want Pewdiepie merch." "C'mon dad, c'mon dog, hook it up!"
Yuriy Yarovoy: It's amazing. We talked about this yesterday, right? I think from YouTube alone he makes something like 10 or 12 million dollars a month. It's stupid. Like stupid money. I'm very happy for him, he does a wonderful job, but to bank ... And that's not including endorsements or anything, obviously, or that merch. But to make that much money from making YouTube videos, it's pretty wild.
Noah Learner: Okay. This is my [crosstalk 00:49:15]
Yuriy Yarovoy: So anyway, we built out a platform that basically services the creators, right? They select what game they want to promote, to content moderation, to affiliate link generation, all that is handled within the platform; also payments, obviously. So, reporting and payments. The Streamer Academy, what you see is, we had a product in BlueStacks called BlueStacks Live where we actually tied in Facebook Live and Twitch, and you can, with one click, actually just stream from BlueStacks to these platforms. As part of that we wanted to kind of, rather than go directly to the big fish and say like "Hey, we wanna work with you, and you promote our product and you get per install, that business model."
Yuriy Yarovoy: We wanted to build that pool of streamers ourselves and empower our community to actually start creating content for themselves, and obviously as a byproduct of that promote us; because that's what they're streaming off of. So we created this entire section of just resources, and we used Coupa and ... Oh my goodness! Oh, Monster ... Oh, man. I forget his name.
Noah Learner: I noticed that we announced that you were gonna be on the show, he was the first person to like the tweet.
Yuriy Yarovoy: Yeah. Coupa's awesome. We have a great relationship with him. We've worked with him a lot in the past. We get a lot of feedback from him, especially regarding our affiliate platform. He's an awesome guy. Very, very happy that we've gotten to work with him as much as we have. So he's been a huge help in actually shaping how we approach influencers, or how we approach creators in general. Just because this is his lifeblood. He created a bunch of content for us, about like how to stream, how to become a streamer.
Noah Learner: Can you see my screen?
Yuriy Yarovoy: Yes, sir.
Noah Learner: This is my favorite question to ask the prospective clients. What keeps you awake at night? Just so I can learn about their pain points.
Yuriy Yarovoy: Yeah.
Noah Learner: Let's talk about yours.
Yuriy Yarovoy: Sure. Is it ...? I forget. Paul Graham, I think, says "Do things that don't scale first". So that's kind of what we talked about, right? For automation. You start with the things that don't scale easily. You figure out what those pain points are, and you automate as much of that as possible. But it still sucks. It hurts, right? What keeps me awake at night is the pain of things build scale. Also, management overhead I think is a byproduct of like we had two choices, we could've outsourced ... Or three choices really, right? We could've hired some agency and paid some ridiculous markup, no offense to agency people, to run our campaigns, and they probably still wouldn't be able to do it at the scale that we needed. We could've hired 100 people to do this, which also is not economical, or we could've automated it, right? So I try to let the machines take human jobs as much as possible where it's better to give the machines the jobs.
Yuriy Yarovoy: Sorry, world. Kind of that leads to the next point, right? Like head count for the sake of head count. Negative ROI I think speaks for itself. You're gonna do a lot of things that will lose money, and on the way to making money. That's fine, but things that are consistently performing poorly you should just nix. Don't become emotionally attached. Just follow the data.
Noah Learner: How long does it take you to make that decision [crosstalk 00:53:15] to be statistically relevant?
Yuriy Yarovoy: For us, days at most, because of traffic. But obviously that number's gonna be a function of how much traffic you can drive to any given experiment, any given page, or campaign, whatever it is.
Noah Learner: Wow.
Yuriy Yarovoy: Brand versus UA, User Acquisition, I generally skew towards performance. So when people talk to me about like "Hey, let's run TV." I'm like "Uh, okay, tell me how you're gonna measure that." They're like "Well, we're gonna look at the GRPs, and we're gonna look at uplift during the same time that the TV campaign ran." I'm like "Yeah. No, too fuzzy." But it definitely has its place. We talked about this actually yesterday, display. Display is a wonderful example of this. Search in and of itself is limited by interest and search volume, right? At it's very core, if nobody is searching for pink elephant bubblegum machines, like "Sorry. That query doesn't exist. The volume doesn't exist", and search in and of itself is not going to be a market maker, right? It's very difficult to change that consumer behavior and make them start searching for whatever I just said about purple elephant bubblegum machine. I think that's what I said, right?
Yuriy Yarovoy: Anyway, right? It's-
Noah Learner: That takes us into our one last thing, doesn't it? Our one more thing moment?
Yuriy Yarovoy: Sort of.
Noah Learner: I've got more time.
Yuriy Yarovoy: Yeah. But basically in short, to keep my rants short. Search is not a market maker. Whereas display can actually bleed over time, or if you spend enough money on creatives and iteration you can actually introduce ideas to people. Search captures people after they've already had an idea, display can capture people on the way to an idea. So that's incredibly powerful. Same thing with TV, right? TV can introduce concepts and ideas, but measuring that is really, really difficult. At least display I can measure. Anyway, okay. Cool. One more thing. Are we not doing the Steve Jobs sign off and then one more thing?
Noah Learner: Oh yeah, sorry.
Yuriy Yarovoy: Aw, dropped the ball, man.
Noah Learner: I'll pretend I'm walking away. Thanks, guys. This was awesome. Okay, your turn.
Yuriy Yarovoy: All right. This is just like something that I was actually talking to a friend about this, and he was looking at ways to prospect content partnerships. He works for a company that I'm not going to name, out of New York, for a beverage that you drink regularly. That is as broad as I'll get, and it's not water. Although that is a component. I guess it's a component of any beverage, right? Anyway ...
Noah Learner: Hot or cold?
Yuriy Yarovoy: Yeah. Yeah. Anyway. So I think one of the most powerful and under-looked places to look for either traffic partnership, or native advertising placements, or influencer campaigns, if you run display ads, especially in GDN, looking at your placements report is hugely powerful. Because one, if you're running display already, you know how each of these placements perform for you, right? For instance ... Can you zoom in a little bit?
Noah Learner: Hm, let's see if I can. Hold on a second.
Yuriy Yarovoy: Yeah, that'll work. There you go. So, Android Authority in the right hand table, I know how well my placements on Android Authority work. I also know that AdSense, especially for larger publishers, they get the worst placements possible. If they have native ads they're gonna give their own native ads the preferred placement on each page. If I know that's my AdSense placement on their site it's performing at certain ROI, or a ROAS. Generally we run our display on ROAS. I know for a fact that if I go to Android Authority that their audience is already interested in my product, and their native advertising's gonna perform twice, three times, 10 times better than their AdSense ads.
Yuriy Yarovoy: So right there I have significantly better performance with just a quick conversation. On the YouTube side, for running video ads, or even just text display ads on the sidebar. I know what channels are actually converting really well, and now I can go out and talk to these people and say like "Hey, you wanna join our affiliate platform? We have a huge amount of offers, you guys can create content, you get paid per install. Great, everybody wins." It's just a very, very, very easy way to prospect, because you've already done most of the work by the time that you get to this report. It's just a matter of having that 2nd conversation, or that next conversation.
Noah Learner: Can you go into costs per click and how they differ, like if they offer both AdSense spots, and also native?
Yuriy Yarovoy: Yeah. Some publishers will only allow you to run CPM, but at that point if you're already running display you know generally what click rates have been on the site. You also know what your return has been. Again, because you're running CPC on the site, great, that's what you're paying for, but you can generally apply that same CPR to their native ads and at least get some idea of what your click through's gonna be per 1000 impressions.
Noah Learner: Do you also think about "Okay, I know that I'm ..." whatever, "500 pixels higher up on the page. I expect to get a higher click through rate." Do you have formulas for that, or are you just-
Yuriy Yarovoy: That's a little harder just because it's gonna vary publisher by publisher. Also it depends on mobile traffic. For instance, let's say CrackBerry, that's an interesting domain. Let's say CrackBerry, like 95% of their users are on mobile. For me they're not really relevant, and if possible I would ask to be excluded from all mobile traffic just because for a desktop product there's not a clear conversion path for those users. But if you're a mobile product and you can get top of the page placements on mobile, like you're mobile bold, so to speak, that's awesome. But I don't think there's a hard and fast rule for "Well, if you're 600 pixels below the top header, you always get three percent click through." I don't think that exists.
Noah Learner: Nice. This was great. Anybody got questions that we want to ... Anyone wanna jump in? Anyone wanna talk? I'm un-muting everybody whose still with us. You guys got any questions? Yuriy, this was kick-ass. I think this was really good. I think the format was pretty smooth, and I'm really stoked. Do you have any questions? Any feedback? Do you have a sense of what your next challenge is? What are you thinking about next that you can share?
Yuriy Yarovoy: I wanna hit that 80% of people that haven't done the math. That's always the challenge, and that kind of like what I'm always going after. Whether that's via some automated means, or doing the hard work until we figure out if it's automatable, that's a different conversation. But that's my challenge.
Noah Learner: Yeah. Well, I gotta say, I'm really, really stoked, and I'm just really glad that you could make it today and share a lot of insight with us. I learned a ton both in talking with you yesterday and today, and I look forward to ... Are you gonna be MozCon? I'm really stoked to see you there.
Yuriy Yarovoy: Yes, sir. I'll be there.
Noah Learner: Okay. Badass. Well, everybody, thanks so much for joining us, and until next time we're gonna be back in two weeks on April 19th. We've got Sam Marsden from DeepCrawl. We're gonna be talking about all kinds of automation stuff. An area focus for him has been a lot of automated reporting types of tasks. So I'm really stoked to dive in with him. Yuriy, I'm so grateful you came, and I'm really stoked to see you soon. Everybody-
Yuriy Yarovoy: Yeah, absolutely.
Noah Learner: ... have a great day, and thanks so much.
Yuriy Yarovoy: Yup. Thanks, guys.
Noah Learner: Ciao.
Yuriy Yarovoy: Bye, everyone. Bye.
Share and learn about automating your digital marketing agency.