(#37xr3ra) As a quick experiment, I modified my code to remove the domain restrictions and low and behold:
```
All done!
Crawled 516 feeds
Found 52464 twts
Found 736 feeds
```
The Twtxt network is larger than I thought. A significant no. of feeds no longer work obviously, but that's okay, we can prune dead feeds out.
matched #4vuxgtq score:11.2
Search by:
Search by 1 tags:
@prologic (#37xr3ra) in theory shouldn't need to let users add feeds.. if they get mentioned by a tracked feed they will get added automagically. on a pod it would just need to scan the twtxt feed to know about everyone.
matched #3antibq score:11.2
Search by:
Search by 1 mentions:
@prologic (#37xr3ra) in theory shouldn't need to let users add feeds.. if they get mentioned by a tracked feed they will get added automagically. on a pod it would just need to scan the twtxt feed to know about everyone.
matched #ij3lzwq score:11.2
Search by:
Search by 1 mentions:
@prologic (#37xr3ra) sounds about right. I tend to try to build my own before pulling in libs. learn more that way. I was looking at using it as a way to build my twt mirroring idea. and testing the lex parser with a wide ranging corpus to find edge cases. (the pgp signed feeds for one)
matched #hm6yihq score:11.2
Search by:
Search by 1 mentions:
@prologic (#37xr3ra) sounds about right. I tend to try to build my own before pulling in libs. learn more that way. I was looking at using it as a way to build my twt mirroring idea. and testing the lex parser with a wide ranging corpus to find edge cases. (the pgp signed feeds for one)
matched #n7dn5aq score:11.2
Search by:
Search by 1 mentions:
(#37xr3ra) Wait... So you actually wrote a more elaborate crawler without taking a shortcut like I did using colly (_not that it really helps much_) Hmmm? ๐ค Can we take it a bit further, make a daemon/server out of it, a web interface to search what it crawls using bleve and some tools (_API, Web UI_) to let people add more "feeds" to crawl? ๐ค
matched #acie3ca score:11.2
Search by:
Search by 1 tags:
(#37xr3ra) @prologic the add function just scans recursivley everything.. but the idea is to just add and any new mentions then have a cron to update all known feeds
matched #mrtyq7q score:11.2
Search by:
Search by 1 mentions:
(#37xr3ra) @prologic the add function just scans recursivley everything.. but the idea is to just add and any new mentions then have a cron to update all known feeds
matched #wkkru4a score:11.2
Search by:
Search by 1 mentions:
(#37xr3ra) @prologic yeah it reads a seed file. I'm using mine. it scans for any mention links and then scans them recursively. it reads from http/s or gopher. i don't have much of a db yet.. it just writes to disk the feed and checks modified dates.. but I will add a db that has hashs/mentions/subjects and such.
matched #e5kep3q score:11.2
Search by:
Search by 1 mentions: