How does one wait until all of the Javascript is loaded before curling a website? I am trying to download the HTML from one of my pages, but it fetches information asynchronously, so cURL fetches a half loaded page. Is there a way to get cURL to fetch a fully loaded page?

How does one wait until all of the Javascript is loaded before curling a website? I am trying to download the HTML from one of my pages, but it fetches information asynchronously, so cURL fetches a half loaded page. Is there a way to get cURL to fetch a fully loaded page?

• php
• javascript
• curl

Share Improve this question Follow asked Jul 3, 2012 at 17:00 egidraegidra 9,0972121 gold badges6767 silver badges9292 bronze badges 2

• cURL cant process javascript. – Shubham Commented Jul 3, 2012 at 17:02
• cURL does not execute Javascript. It will load the initial document served by the web server and nothing else. Any Javascript that is executed to modify the DOM will have no effect on what you are able to load with cURL. – DaveRandom Commented Jul 3, 2012 at 17:03

Add a ment  | 

1 Answer 1

Sorted by: Reset to default 6

You need to use a headless browser engine to do this. cURL and wget are HTTP libraries; they speak HTTP and download documents as text. They don't have a concept of a DOM or a JavaScript engine that would help them understand that a page is doing AJAX. So to download the HTML, you need something that acts more like a browser, by parsing a DOM and executing JS. I remend Crowbar, which uses a Mozilla engine.