tag:blogger.com,1999:blog-4503292949532760618.post8047771686867693477..comments2024-03-28T13:39:27.601-07:00Comments on DSHR's Blog: Re-thinking Memento AggregationDavid.http://www.blogger.com/profile/14498131502038331594noreply@blogger.comBlogger1125tag:blogger.com,1999:blog-4503292949532760618.post-91311577638052834582013-10-01T07:06:48.743-07:002013-10-01T07:06:48.743-07:00We recently had a twitter conversation about using...<a href="https://twitter.com/geckomarma/status/384655483040448512" rel="nofollow">We recently had a twitter conversation</a> about using some flavor of DHT to distribute this kind of data – and perhaps either adapting something like <a href="http://www.coralcdn.org/overview/" rel="nofollow">CoralCDN</a> or a generic data-store like Cassandra.<br /><br />One of the ideas I've been intrigued by is whether you could simply use BitTorrent to basically publish the list of domains archived by a particular organization, making it trivial to distribute very large lists and allow new players to volunteer their content without a gatekeeper. Obviously you'd still need to tackle the question of trustworthiness but something as simple as a ZIP file containing a text file listing domains and a PGP signature would make it easy to allow users to decide who to trust and still make it very straight forward to protect against a single point of failure. It'd also have the nice characteristic of making minimal demands on the implementation stack and bringing a lot of battle-tested network code, with significant precedent for browser integration (e.g. Opera) which would be really nice for closing the final loop by allowing clients to deeply integrate Memento support.<br /><br />(There's some precedent for deeper integration into a server stack, too: Etsy famously uses BitTorrent to distribute Solr indices more efficiently: http://codeascraft.com/2012/01/23/solr-bittorrent-index-replication/ This could be really interesting from the perspective of how quickly one could bring a new TimeGate server online.)Chris Adamshttps://www.blogger.com/profile/06132981126518774942noreply@blogger.com