(#3ll4fja) I feel like most feeds I would want to subscribe to are updated pretty infrequently so once a day would be sufficient. But I’ve seen other feeds that are updated more frequently. Perhaps some sort of heuristic based off the feed itself? (For example, average time between entries with a max of 1 day.) If you combined that with a feature to manually trigger a feed recrawl, that might be good enough.

(#3ll4fja) > How often would you want your feed(s) re-crawled / re-scraped ? 🤔 cc @xuu @antonio @dooven @jlj @darch @adi @mckinley as the most opinionated Yarn.social users 🤗

(#3ll4fja) @brasshopper This is a good suggestion! Right now the crawler is a "one-short" thing (_just to be nice while I develop/improve it_). From your suggestion it _sounds_ like I can possibly remove the "one-shot" restriction and just setup a daily job that re-crawls the entire Twtxt space from a seed feed. The _best_ seed so far is _probably_ my own followed by @jlj 's -- Then I can focus on the refetch/rescrape parts based in heuristics. What do you think?

(#3ll4fja) > How often would you want your feed(s) re-crawled / re-scraped ? 🤔 cc @anth @dbohdan as the most vocal Twtxt users.

(#3ll4fja) Once a day.. though if it hasn't updated in n-months maybe once a week?

@jlj (#3ll4fja) There are some benefits to more frequent crawling / scraping 🤔 Auxh as filling in missing in a conversation where you don’t follow some participants, global timeline, etc

@prologic @dooven @darch @adi @mckinley @xuu @antonio (#3ll4fja) Yeah, infrequently is fine with me (e.g., once a day).

(#3ll4fja) As a user, I think 1-3 times a day would be fine. As someone who pays to host a twtxt feed, I don't mind what you set it to as long as it's not unreasonably often. It won't really make a difference to me. As of about 6 months ago, 3 different yarn.social pods each request /twtxt.txt (with an If-Modified-Since header) every 5 minutes, 24 / 7. I think I also had a different twtxt client requesting it every 10 minutes.

(#3ll4fja) Once a day.. though if it hasn't updated in n-months maybe once a week?

@xuu (#3ll4fja) What about the utility of rescraping more active feeds so we can do things like fill in missing pieces in a conversation of participants you don’t follow?