Wait… So you actually wrote a more elaborate crawler without taking a shortcut like I did using colly (not that it really helps much) Hmmm? πŸ€” Can we take it a bit further, make a daemon/server out of it, a web interface to search what it crawls using bleve and some tools (API, Web UI) to let people add more β€œfeeds” to crawl? πŸ€”


#acie3ca
This is twtxt search engine and crawler. Please contact Support if you have any questions, concerns or feedback!