Google vs. JavaScript: What is the Score in 2019?

 

 

Read the Transcript

Martin Splitt: Word of mouth, ads, okay, that’s two ways of getting people on the website, you can also sit at a computer and just go on your website or something, I don’t know.

Another way is actually search engines. So you wanna make sure that search engines find your content online, and there’s like a whole profession around that, basically SEOs. Search engine optimization experts who are helping you to make that happen. And there’s many experts that are basically looking at the basic things, and also some are really going into lots of details, but there’s a whole category of things that can go wrong if you’re using JavaScript and not that many people are addressing that.

So I’m really, really happy that we have our next speaker here, named Bartosz, who is being – constantly challenging my life, basically. By basically constantly running tests and going like, “Hey! Quick question! Is this expected behavior that we should see from Google search?” And I’m like, “Yes. Can you explain it to me? Oh, wow, OK!”

So he asks me the really, really hard questions, but he’s a really, really good presenter in answering the hard questions.

So please give it up for Bartosz.

Bartosz Goralewicz: Thank you so much, Martin, for such a nice intro. So we’re most – usually challenging Martin or John [Mueller], so that’s – most of our job is basically annoying Googlers.

But today I want to talk about how to make your website JavaScript-friendly, SEO-friendly, because I’m not sure if you guys – how many of you work with JavaScript websites? Or just with the – Oh, I didn’t expect almost everyone.

So that’s very nice. So basically what most people don’t know about is that most websites, maybe not most, like why do websites have massive issues with JavaScript? And Google are doing so much better, but they are still struggling with a couple of things.

So they want to talk. I’m gonna have a few slides that I’m gonna – we’re gonna ask Martin to close his eyes. But today when I talk about JavaScript and SEO, we all know that this relationship is very, very complicated.

It’s mostly because JavaScript is very, very dynamic and there are so many different flavors, types of JavaScript, and it’s basically an ever-evolving topic.

And if that wasn’t enough, Google and JavaScript have a very, very, very difficult relationship because Google is, only, or maybe not only, how would you say that? Connected to … connected to Angular, how, like, supporting Angular. So Google is somehow connected to Angular.

Martin (offscreen): We have started the project I believe, and we are maintaining it, but just, just for the record: Angular people are really, really nice people. They come to us and say, “Martin can you help us with this thing?” And I’m like “Nope! There’s the open public forum where you can ask those questions and get the same support.”

Bartosz: So Angular, what’s actually – something that probably Martin wouldn’t say, but Angular was one of the most problematic frameworks.

Martin (offscreen): That’s true!

Bartosz: Which is not something that you would expect and the website Angular.io wasn’t indexed in Google for a few years.

Martin: Yeah.

Bartosz: So a lot of people, like,  in 2016 – we are a technical SEO agency. I’m sorry, I didn’t introduce myself. But we deal with technical SEO for enterprises and for e-commerce and for large websites, and in 2016 we got kind of sick and tired of all the developers and website owners asking us about very, very specific JavaScript SEO things that no one knew about. So there’s that time recently where all the JavaScript madness began, so everyone started developing JavaScript without fully understanding the technology and, at some point, JavaScript became more important than basically earnings and traffic and, yeah, and websites.

And JavaScript SEO is still completely misunderstood in 2019. A lot of developers or SEOs are trying to do JavaScript SEO but what they do is basically they advise everyone to prerender their content.

So it’s not really high-end technical SEO or JavaScript SEO.

And if you look at websites like Wish.com – one of my favorite websites. If you disable JavaScript, their website disappears, which is fine, but at the same time they are not indexed in Google with most of the products and content. So they’re walking away from quite a lot of income, quite a lot of money, and this is not the only example. We’re time-limited but we have a database of probably like 100 websites, including, like, Aliexpress, EA games, and all the big brands that lose massive traffic, and I’m not talking like 10-20% but sometimes like 80% because of how they handle JavaScript.

If you look at Wish.com and their visibility chart, you can see that this relationship is not going well.

This is like an equivalent of a couple that’s having a very rough time. They get together and then it’s happening again. So this is more or less what’s happening with quite a lot of big brands and JavaScript SEO. And there is a massive cost of JavaScript, a massive cost that somehow is overlooked.

Just to show you an example, let’s have a look at the video of CNN being opened on iPhone X and Motorola G4, just to show you how resource-heavy JavaScript can be. So I’m not gonna play the full thing because the Motorola G4 crashed halfway through, but if you look at the iPhone X, it loaded CNN, but Motorola is still struggling to even send the request, so this is a problem that we have to face.

And desktop is dead.

And this is the data from Germany, and if you look at Switzerland it’s actually very, very surprising because if you look, again, Germany 23% for Google.de comes from desktop. So just 20% of traffic comes from desktop devices right now. In Switzerland this is shockingly – you guys still have quite a lot of desktop traffic which I totally don’t understand.

Like, in Europe you’re, like, actually one of the exceptions. But most countries, you guys, are not – most countries actually don’t have that much desktop traffic any more.

Let’s move back in time to 2015 and look at some of the data. So Hulu.com was one of the websites that actually inspired us to dive deeper into this topic. So Hulu.com if you – do you know Hulu.com? They’re the biggest competitors of Netflix and we will see in a minute why there are not doing that well recently. So there – they actually jumped onto the JavaScript hype train in 2015-ish and they figured, okay, they were very curious about that but they couldn’t figure this out.

They didn’t ask themselves one of the most important questions: is, like, can this backfire somehow? Can launching a JavaScript SEO website just – sorry, JavaScript website without full knowledge of JavaScript SEO, is a good idea?

This is – this is how their traffic looked after that. They’ve lost overnight – they lost more than 40%, then within the next upcoming weeks, they’ve lost all together around 70% of the traffic. It goes so bad that you couldn’t find any of the Hulu shows in Google and they were only available at Hulu.com. So if you want to watch any show from Hulu.com the only website that would show up in Google were torrents.

This is how bad it looks like and it’s still not fully fixed until today. But let’s move forward.

So what exactly went wrong? There was no data at that time and we figured, okay, if there is no data then we need to get it somehow. I need to figure we’re gonna create a dream team to basically build an experiment.

And it was the first JavaScriptSEO experiment in the history of the internet. We chose this very cool, very cool name. So it was JSSEO.expert.

The idea was extremely, extremely geeky. A simple query – they were very, very simple. We basically built a website where every single subpage is a different framework. So we would see, OK, how is Google, and other search engines we’re gonna talk about that in a second, how are they dealing with indexing all the different frameworks?

So basically if you go to JSSEO.expert and /vue, you’ll see content generated by Vue. It couldn’t get any simpler. At the very beginning we only had “Hello world!” in, like, within a few days we just – we’ve found a way to generate quite a lot of content through machine learning. But we basically went with Hello World and that wasn’t available in all of the framework, and some of it was, but basically just to give you a short example of how it worked like: if you use, switch off the JavaScript. It disappears. Very simple.

So if we see that Google indexed the content within the red frame, we know that Google parsed that JavaScript framework, the JavaScript code. The idea was extremely, extremely simple, but the results were really, really mind-blowing.

If you look at that Google did pretty well with most of the frameworks, it actually struggled a little bit with Angular – it struggled with AngularJS because it turned out it wasn’t Google’s fault. It was Google’s AngularJS’s fault because they had a problem with their own library. But it got quite interesting because it turned out that Google didn’t crawl the code that was placed externally, but they crawled the content that was based – if the JavaScript code was in line it was working fine.

And we found quite a lot of different anomalies. We played with the data and we basically found a video from 2015 from Jeff Whelpley, who’s somehow involved in the Angular project, saying that if you care about SEO, you still need to have server-rendered content. And this got us thinking how our other sections was dealing with that. And if you look at that, only Google and Ask. Ask is not saying that publicly, but Ask is using Google’s servers and basically database, so Ask is just simply Google. It’s not publicly announced but we fight with that, and it is.

So only Google is able to crawl any JavaScript as of 2017. It changed a little bit. I was talking to the guys who were, like, involved in that project within the Bing – JavaScript project for a while but they are doing something with JavaScript now – is still not even close to the work that was in 2002, but this still wasn’t the main problem. Because we figured, OK, Google can index some of the content, is tricky, but it wasn’t the main problem. We couldn’t find any JavaScript website that ranks until 2018.

And I reached out to John Mueller – good questions – and John Mueller, at that time, unfortunately, couldn’t help us for reasons we can only speculate about. What I’m guessing, they basically didn’t communicate their problems of JavaScript or, I don’t know, but John Mueller couldn’t really help us with our questions. So we figured that we’re going to build more experiments.

Before I move forward with that, I’m guessing you’re pretty technical so I’m not sure this slide is important, but just for those of you who aren’t: HTML is just like a ready-to-go cake, so all the bots, all the users, just basically get the content they seek.

With JavaScript, it’s a little bit more complex, you get all the pieces that you should render – or in this scenario – bake together to get the final four things. It’s very, very, very expensive. So looking at that, some frameworks are indexable, but there is a very, very fine print.

It seems that Google is not really, really fond of JavaScript but at the same time, they are very, very interested in HTML. And if you wanna  . . . meme, basically, we saw that every single HTML website is indexed within minutes, with JavaScript it’s not that easy.

And I’m guessing you already know where I’m going with that. But we’re going to build one more experiment. And we’ve created the simplest possible experiment we could think of. We created a very simple JavaScript and HTML website – they’re both identical – but you can only get to page number 3 from page number two and so on.

So basically, if Google would index the page number six we knew we knew that they basically went through every single one of them. With HTML, Google will index them within minutes. With JavaScript, from the home page to page number one and it completely died. And it’s not even – I’m not even talking about hours or days but Google didn’t index any JavaScript within that page for six months.

Basically, everything after – after the first sub-page was invisible. And we’ve been repeating this experiment actually until today. There is a few things that has changed, I’m gonna talk about that but around 2017 and the results are always the same. Google would never crawl JavaScript deeper than the homepage and the first linked page.

We came to the conclusion that JavaScript, even if indexable when you force Google to do that, is gonna kill your crawler budget completely.

After roughly a year, John Mueller admitted that crawling and indexing is slower for JavaScript than static HTML.

We’ve been actually waiting for that for – for two years, from like – the main problem we had is that developers didn’t really believe us, even though we had experiments – because we are the only ones talking about that. So this I don’t even remember that was a tweet or a hangout, this sentence from John Mueller is something we almost tattooed on our foreheads for over a year because we finally had something to talk about with developers. And his advice helped in general, so he’s – his help was really appreciated.

And I know it works both ways, I know that Google actually used that experiment as well. So after failing at the beginning with John Mueller, I reached out to most, like, like, we actually reached out to, like, 20 different Googlers, but Ilya Grigorik was the one who responded. A lot of them were emails, tweets and whatever. And we invited Ilya Grigorik to our Google Doc with every single thing we found out and he started commenting, commenting like crazy for a day, and then he disappeared.

But he said that JavaScript SEO – JavaScript is not bad for SEO if done right, and he mentioned that there is a cost coming from basically rendering that at the Google side. So if you look at JavaScript – JavaScript if it’s not done right, which is a very broad statement. It’s very expensive to run and that’s something we actually confirmed. So even if we put any, like, random number you can see that HTML is really cheap, like, let’s say a page in HTML cost $1, for JavaScript it’s gonna be, let’s say, $100.

Someone offscreen: Scale!

Bartosz: Yeah, so fast forward to 2018, and one more thing from Google that we really, really enjoyed is that John Mueller had this Google i/o talk . . . and they mentioned finally that Google is indexing JavaScript with two waves. So the first wave is basically they only look at HTML and the second wave is when the resources are available – is indexing HTML. Sorry, JavaScript. Yeah, and this actually got a little bit tricky because JavaScript and the second wave can take up to – they’ll – like Googlers said up to two weeks.

In our experiment, in our experience, it can take up to forever, which is – So we have websites from 2016 that are still not indexed.

Let’s go back to Hulu for a second because this is still ongoing. Hulu after, like, one year from their massive drop, figure, OK we’re gonna fix that. They fixed the problem to JavaScript on the desktop version. What they didn’t know (and that’s actually the best jokes I have in this deck) the mobile indexing went live around the same time. So they fix it for this and Google never saw that.

Just to show you it’s something that’s extremely, extremely interesting here is that – Oh! So we’re not gonna finish on time. We have 100 slides and, yeah. So mobile Google bot and basically Chrome looks at the content from a different point of view, at that time, because they were using Chrome41. And we saw something that’s called Casual Indexing – actually, I came up with that after we found that first.

So all the content that wasn’t available for mobile Google bot wasn’t indexed. So if you see that was available only for basically browsers newer than Chrome 41, and it wasn’t available for mobile Google so this comment wasn’t indexed at all, but at the same time – I’m sorry this is like geek humor – at the same time, if you look at anything that’s available in both those screens, is fully indexed.

They indexed only the static page of their website, which is hilarious to me because it took them like one-half year or something like that . . . But I’m guessing it’s not funny for Hulu. Anyways, any content here, again, wouldn’t be indexed in Google. So they indexed just part of their website.

Fast forward to 2019 because we’re running out of time – Google got a little bit better with crawling and indexing JavaScript. So Google actually bulked up a little bit.

I’m guessing they either built new servers or hired more developers. I have no idea what they did.

We probably need to get Martin drunk and find out. By the way, it’s every SEO’s dream just to get a Googler drunk. It’ll never happen. So we’re still waiting, Martin.

[Martin speaks offscreen]

The crawler budget repeatedly in 2019, if you remember the explosion here, is gone. So basically the test websites we’ve created a few months ago, a few weeks ago . . . they all work.

So Google indexes JavaScript right now almost instantly, but there are some exceptions and we’re going to talk about those.

If you look at National Geographic, they’re a very good example: if you switch off JavaScript the content disappears. 100% of JavaScript content indexed quickly because we checked that as well. We looked at the sitemaps . . . all nice.

As for Asos.com: the same story – without JavaScript the content disappeared. 100% indexed.

But there are some things that don’t work yet and this is the part that’s actually gonna be a little bit upsetting for Martin I’m guessing. And – but there is some percentage of JavaScript content not indexed for large websites and this is actually where I’m getting with that.

This is quite shocking. If we look at the random samples, and this is not for like the whole content, but just the content that’s generated by JavaScript: Vice.com – 74%. For example, Cosmopolitan.in, if you look at that they only indexed 49% of the content and everything that relies on JavaScript is not indexed.

Again, random sample, the system likes new content – it’s just like that. So if two waves of indexing – there’s one more thing, as I mentioned before you need to expect some delays indexing your content and this is the most shocking slide I have in my deck.

I’m guessing some of you will get that. We will see that as extremely interesting here. So that’s a geek test. If you find it interesting, sorry.

And so a percentage of the JavaScript content not indexed after ten days. So we look at the sitemap, we take, like, a sample 100-1000 URLs and we check how much content is not – is still not indexed after 10 days.

So looking at like Channel 4 television, none of their JavaScript content is indexed after four days – ten days! You would expect that to be quite efficient for, for a website this size, with this authority.

Allrecipes, 14% is also quite good. But, for example, I thought 50% of the content is still indexed after 10 days.

But we’re going to talk about a little bit of black hat, and again, I’m super sorry, Martin. This is not like a link-building black hat, this is geeky black hat. So cloaking 2019, this is something that we find extremely interesting. And we have a few websites, we have like Pepe, we have guns, we have Trump.

Today we’re gonna talk about guns because I know that you guys have guns in Switzerland. So we use this example. This is actually a website that’s cached by Google. So Google sees, OK, gun control – we’re all for that.

But if you open that website in your browser you’re gonna see completely different content.

The only thing we did here is that you have to render JavaScript to see that. So you have a massive, massive potential – this is what you see in Chrome, this is what you see in Google cache.

And this is not only about the yes or no, but the whole content is completely different.

We can – I’m not advising that, but it’s possible to play with content and hide content from Google with JavaScript, if basically JavaScript is gonna rewrite the HTML. This is a little bit shady – I don’t invite that but –

So, just to finish, like the final statement: Google got extremely, extremely good with crawling JavaScript and they are the only search engine right now doing that at scale. But if we look at that from our perspective, we’re working with some of like the biggest e-commerce stores worldwide with the new game and new rules. There are new players, basically winning and losing just because of JavaScript. Just to show you the example of Netflix vs. Hulu: Hulu was much bigger in the States for years.

So Hulu was winning until 2016 when, when Netflix basically took over because Netflix – if you do not know, probably some of you are aware of that – Netflix is extremely good with technology, with JavaScript. They’re like thought leaders in that field. Hulu, not so much. They’ve been bought by Disney,  and some of my friends from Disney actually working with Hulu now, and I’m hoping this is gonna get better eventually, but if you look at that, Hulu lost just because of JavaScript and how – probably there were some other problems, but mostly because of JavaScript, Hulu lost their position on the market.

One last thing this is my second last slide – almost.

If you look at that . . . I have this theory that’s, yeah, I have this theory that once Google is gonna completely index and, like, gonna be extremely good with indexing and rendering JavaScript, Bing is gonna stand no chance because if all of you are gonna publish your websites in JavaScript, Bing is never gonna afford, with the market share they have, they will never afford rendering and indexing JavaScript at scale, because of the price. So once they do that – and this is the animation I did myself . . .

So once they do that Bing is gonna be a massive, massive problem – they’re gonna actually struggle with that. Suggested to do for you, because I believe that everything should be actionable if you’re choosing a framework please leave these don’t go with, for example, Knockout, because we have clients who implemented Knockout a year ago, if you look at Twitter, the last tweet is from three years ago, so we can assume that this framework is completely bad.

Look at the frameworks that somehow support server-side rendering and/or some of the new, new solutions in the JavaScript field. Then don’t push client-side rendering content to Google because it still it works in most case. but it’s still not perfectly. It’s not gonna be for a while, I’m guessing, so if you choose in any way to process your JavaScript, client-side rendering, this is the least – this is not, this is the best year.

So client-side rendering is the worst, go with hybrid rendering or server-side rendering – pre-rendering, if you really, really have to. It’s not really very good. At last, whatever you do with JavaScript, you need to experience – experiment and measure before deploying your website so you don’t do what Hulu.com did. And that’s it. Thank you so much.

Martin: Bing is a fine good search engine. It’s not like they’re – we’re actually working a lot of with Bing because fundamentally we want the same thing . . . we all want the web to win and they’re doing fantastic things. And I’m not so sure that they don’t catch up quickly and they’re actually doing great stuff behind the scenes.

Browse the Deck