The answer to this question is rather complex and could just as well be a separate article. To simplify this topic, let’s just say that it is all about computing power. With HTML (PHP, CSS, etc.) based websites, crawlers can “see” website’s content just through analyzing the code.
Basically what you see above is code “processed” by the browser.
To see how the source code looked like before rendering you need to use “View Page Source” option.
After doing so, you can quickly notice, that all the content that you saw on the page isn’t actually present within the code.
After setting this up, we can start crawling data and see each page rendered.
Word of warning
Let me show you an example – as you saw above, Screaming Frog properly crawled and rendered this URL: www.hulu.com/casual. However, this URL isn’t properly indexed by Google.
Here is proof. Google cache:
Let’s copy and paste content from the Casual TV show landing page:
Unfortunately content from this page isn’t indexed in Google.