(#momapxa) @mckinley cause of probably write a command-line tool for you that uses that library. If you tell me what basic things you need it to do?

(#momapxa) @mckinley Wait what?! ... Why?! What are you doing? Are you trying to write a web search engine / crawler? 😅

(#momapxa) @prologic I'm trying to make a static local mirror of MDN Web Docs. It's all free information on GitHub, but the whole system is extremely complicated. <​tinfoil-hat>I think it's so they can sell more MDN plus subscriptions, making people use their **terrible** MDN Offline system that uses the local storage of your browser.<​/tinfoil-hat> At this point, I'm willing to run a local dev server and just save each generated page and its dependencies. I really only need it to run JavaScript so it can request the browser compatibility JSON. It's but the MDN server, annoyingly, transforms it. Once the BCD is rendered statically, I should be able to remove the references to the JavaScript. That will solve another issue I'm having where the JavaScript is constantly trying to download `/api/v1/whoami`, which seemingly has no purpose aside from user tracking.

(#momapxa) @prologic It's close, but it's just a Web scraping library. I'm looking for something of the command line variety.

(#momapxa) @mckinley I can confirm the library "does what it says on the tin" 👌 I'll put up my little CLI tool up for you to play with, its pretty damn stupid and basic right now as I'm not completely yet really sure how to flesh this out. Will need you to guide me on this, there's probably a fair few nuances to writing a decent web mirroring tool (at least it does the right thing though and handles dynamic content rendered with Javascript -- Which I tested by hitting my files.mills.io web app which has a pure JS frontend using MithrilJS)

(#momapxa) @mckinley Yes yes but if this library will do the trick happy to write said tool for you and who knows anyone else in need?

(#momapxa) daohoangson/go-sitemirror: Website mirror app with priority for response consistency.