@xuu (#37xr3ra) This is true!

@prologic (#37xr3ra) Cool!

@prologic (#37xr3ra) in theory shouldn't need to let users add feeds.. if they get mentioned by a tracked feed they will get added automagically. on a pod it would just need to scan the twtxt feed to know about everyone.

(#37xr3ra) @prologic It is pretty basic, and depends on some local changes i am still working out on my branch..

(#37xr3ra) As a quick experiment, I modified my code to remove the domain restrictions and low and behold: ``` All done! Crawled 516 feeds Found 52464 twts Found 736 feeds ``` The Twtxt network is larger than I thought. A significant no. of feeds no longer work obviously, but that's okay, we can prune dead feeds out.

(#37xr3ra) @lyse @prologic very curious... i worked on a very similar track. i built a spider that will trace off any `follows = ` comments and mentions from other users and came up with: ``` twters: 744 total: 52073 ```

(#37xr3ra) It _might_ be worthwhile combining the two approaches and _actually_ building a goodness to gracious search engine and crawler for twtxt? 🤔 🤣

(#37xr3ra) Wait... So you actually wrote a more elaborate crawler without taking a shortcut like I did using colly (_not that it really helps much_) Hmmm? 🤔 Can we take it a bit further, make a daemon/server out of it, a web interface to search what it crawls using bleve and some tools (_API, Web UI_) to let people add more "feeds" to crawl? 🤔

(#37xr3ra) @prologic yeah it reads a seed file. I'm using mine. it scans for any mention links and then scans them recursively. it reads from http/s or gopher. i don't have much of a db yet.. it just writes to disk the feed and checks modified dates.. but I will add a db that has hashs/mentions/subjects and such.

(#37xr3ra) Ahh I don't think your code actually _crawls_ the Twtxt space right? Just parses urls given to it and adds it to a database file?