I feel like most feeds I would want to subscribe to are updated pretty infrequently so once a day would be sufficient. But I’ve seen other feeds that are updated more frequently. Perhaps some sort of heuristic based off the feed itself? (For example, average time between entries with a max of 1 day.) If you combined that with a feature to manually trigger a feed recrawl, that might be good enough.


#az2niga
This is twtxt search engine and crawler. Please contact Support if you have any questions, concerns or feedback!