@prologic (#37xr3ra) sounds about right. I tend to try to build my own before pulling in libs. learn more that way. I was looking at using it as a way to build my twt mirroring idea. and testing the lex parser with a wide ranging corpus to find edge cases. (the pgp signed feeds for one)

@lyse @xuu (#37xr3ra) Hmmm very interesting ! Let me put my code up somewhere

@prologic (#37xr3ra) in theory shouldn't need to let users add feeds.. if they get mentioned by a tracked feed they will get added automagically. on a pod it would just need to scan the twtxt feed to know about everyone.

(#37xr3ra) @prologic yeah it reads a seed file. I'm using mine. it scans for any mention links and then scans them recursively. it reads from http/s or gopher. i don't have much of a db yet.. it just writes to disk the feed and checks modified dates.. but I will add a db that has hashs/mentions/subjects and such.

(#37xr3ra) @prologic the add function just scans recursivley everything.. but the idea is to just add and any new mentions then have a cron to update all known feeds

@prologic (#37xr3ra) sounds about right. I tend to try to build my own before pulling in libs. learn more that way. I was looking at using it as a way to build my twt mirroring idea. and testing the lex parser with a wide ranging corpus to find edge cases. (the pgp signed feeds for one)

(#37xr3ra) @lyse @prologic very curious... i worked on a very similar track. i built a spider that will trace off any `follows = ` comments and mentions from other users and came up with: ``` twters: 744 total: 52073 ```

(#37xr3ra) @prologic It is pretty basic, and depends on some local changes i am still working out on my branch..

@prologic @etux @xuu (#37xr3ra) Now I want to remove the "domain" restriction, add a rate-limit and _try_ to crawl as much of the Twtxt wider network as I can and see how deep it goes 🤔

@xuu (#37xr3ra) I _think_ what I have put together last night is a little different... 🤔 -- What you've got is a lot more code and looks way more polished 🤗 At a high-level what does yours do?