News:

"Recent developments in peer to peer networks have centered around the concept of distributed hashtables or content routing. This approach assumes possession of a hash or other identifier that precisely specifies the document the user wishes to retrieve. Naturally there are some situations in which a user only has more general information about their needs, e.g. keywords or other meta-data. A number of different strategies for handling this kind of search in peer to peer networks have recently come to light. A number of them are summarized below along with an attempt to identify some common themes"

...

"Searching naturally divides into first determining the unique IDs of documents that match a query, then finding the actual documents with those IDs. Not all the systems outlined below support this division of labor. However, any system that maps meta-data to document IDs could clearly be used in tandem with one that maps those IDs to actual document locations. "

This is a good point. We don't need to have download support within the search framework. This also allows us to use kademlia, chord (or something not yet in existance) and also swarming technology like BitTorrent.

Recently an article was published that attempted to define the role of aggregators within the RSS world.

"A news aggregator is 'software that periodically reads a set of news sources, in one of several XML-based formats, finds the new bits, and displays them in reverse-chronological order on a single page.'"

"Because Radio's aggregator is integrated with a powerful weblog tool, you can view it as a 'content router.' When you see an item that you want to amplify with your own comments"

... not really. I think it is just a hybrid application (publisher + aggregator). Though 'content router' does seem more catchy.

" But the centralized approach had severe limits, especially as the number of XML-based feeds grew and then exploded. The My.UserLand aggregator subscribed to every feed that was registered. The hourly scans took longer, and users rightly clamored for the ability to subscribe to feeds, to only see those sources that interested them. But UserLand didn't have the resources to build the big centralized system that the users wanted, so of course, we decentralized."

Excellent. This is why it is important to have a decentralized system.

I believe there are three types of main RSS applications.

Producers

Producers are client applications that are used to create/publish new articles and commentary. They generally support the following core functionality.

Examples: Blogger

Readers

Examples: PineApple

Aggregators

Examples: Reptile , Meerkat , DayPop

Hybrid Applications

As usual everything doesn't perfectly fit into nice organized categories. A lot of these applications have features from each main category.

The current version of mod_content is very nice but needs a few more features to be ready for prime-time.

Specifically we need the following features:

I have been doing a lot of thinking recently about building a spider that finds RSS documents on websites.

Given the fact that there is no central repository or search engine for RSS content I think this will be very nice.

Granted we have syndic8 but at the current point there are only about 6000-8000 RSS feeds that can be found through OCS providers. I know for a fact that there are at least 500,000 (livejournal) and blogger should add another 500,000 when/if they push out RSS to all users.

What is really needed is an easy way to spider the whole blogspace and find all RSS links.

Right now we have the little XML icon that is used to visually note an XML/RSS feed.

Users are now providing blogrolling links on their sites. I am sure many of you have seen these. Most of these are driven by CMS systems so they automatically produce RSS.

Currently there is no way to find RSS links without downloading and testing all other links on a site. This can seriously increase the amount of resources needed by a spider.

If RSS producers would just link to RSS with:

<a href=/old?u=http%3A%2F%2Fwww.peerfear.org%2F%3Ci&y=1999>"_url_" type="text/xml+rss">

They can automatically be picked up by spiders.

What do people think about this? Should it be adopted?

I don't think it solves all our problems but it seems to be technically correct and easy enough to implement.

SUPPLEMENTAL: // created on Tue Jun 25 2002 09:06 PM

The consensus on the rss-dev mailing list is that this should be application/rss+xml and not text/xml+rss.

"In the swirling debate over digital rights and intellectual property, Bram Cohen ran into a surprising discovery as he developed a new software program in the past year: It's an uphill battle to convince people that individuals can make their work freely available over the Internet, available for folks to use however they want without any obligations."

...

"Cohen's experience is one small reflection of just how politicized the issue of intellectual property on the Internet has become on every level. The Internet, once lauded as a frontier of freedom and a place for innovation to flourish, has become a battleground in an emerging war over who should own vast swaths of information and the next good idea."

I would like to know how the author found out about the fact that Bram has been struggling with this? I know we talked about it on IRC once, my point being that things should not be PD as long as the copyright hasn't expired. Release under an OSS license.

...

and of course the obligatory mention of the Creative Commons:

"The other is the launch of a nonprofit corporation Creative Commons (www.creativecommons.org), an ambitious effort dedicated to the idea that it's possible to fine-tune the exercise of intellectual-property rights in cyberspace."

"Abstract. Some members of the open-source and free software com- munity argue that their code is more secure, because vulnerabilities are easier for users to nd and x. Meanwhile the proprietary vendor com- munity maintains that access to source code rather makes things easier for the attackers. In this paper, I argue that this is the wrong way to approach the interaction between security and the openness of design. I show first that under quite reasonable assumptions the security assurance problem scales in such a way that making it either easier, or harder, to nd attacks, will help attackers and defendants equally. This model may help us focus on and understand those cases where some asymmetry is introduced."

I don't think it will help them equally. With Open Source we can have more defendants. With closed systems you have the same amount of attackers but less defendants.

Also, what about explicit security holes? Specifically the addition of of flaws that can be exploited by the company or governments that produce/distribute he software.

"However, there are more pressing security problems for the open source community. The interaction between security and openness is entangled with attempts to use security mechanisms for commercial advantage { to entrench monopolies, to control copyright, and above all to control interoperability. As an example, I will discuss TCPA, a recent initiative by Intel and others to build DRM technology into the PC platform. Al- though advertised as providing increased information security for users, it appears to have more to do with providing commercial advantage for vendors, and may pose an existential threat to open systems."

OK... yes. The TCPA seems like it might be very scary!

"Transaction costs may persuade some vendors to remain closed. For example, if source code were made available to beta testers too, then the initial reliability of beta releases would be worse, as the testers would be more e cient. Fairly soon, the reliability would stabilise at the staus quo ante, but a much larger number of bugs would have had to be xed by the vendor's sta . Avoiding this cost might well be a rational argument against open source."

Which is one of the major points. Microsot doesn't want to fix these bugs as it costs them more money. OSS suffers less from this problem because more hackers can look at the source code.

"The Sony Playstation 2 uses similar authentication to ensure that memory cartridges were made by Sony rather than by a low-price competitor - and the authentication chips also contain the CSS encryption algorithm for DVD, so that reverse engineers can be accused of circumventing a copyright protection mechanism and hounded under the Digital Millennium Copyright Act."

...

"If you boot up your PC in an untrusted state, it will be much like booting it with Linux now: the sound and graphics will not be so good, and there will not be nearly so many applications."

ouch!

...

I think the TCPA and Palladium will be a failure with consumers rejecting the technology. History shows that people want open systems. This does not mean we should grow complacent. Join the EFF and fight the DMCA !

"MAGNET is an open URI-scheme and supporting practices/code for enabling seamless integration between websites and locally-running utilities, such as file-management tools. Here's an example of what a MAGNET URI could look like: magnet:?xt=urn:sha1:YNCKHTQCWBTRNJIV4WNAE52SJUQCZO5C"

This is very cool. Gordon Mohr has been doing a lot of good work here. I would really like to see this implemented within M3U and RSS files. Since they are ordinary URIs supporting this in RSS should work just fine.

I an pondering setting Reptile up as a permanent service running under peerfear . It will have a an HTML interface for running searches with a web browser, SOAP interface for running queries from native code, and RSS interfaces for running new queries and getting the latest news.

Most of the code base for this is already complete. I just need to deploy it on the new machine and setup mysql as the database.

Any thought to what the name should be? I was thinking about putting it under reptile.peerfear.org but this doesn't seem like a good idea as I would like to put the main website there at some future date. Maybe aggregator.peerfear.org? Maybe some other name?

SUPPLEMENTAL: // created on Mon Jun 24 2002 04:33 PM

Maybe it should just be home.peerfear.org or reptile-home.peerfear.org?

END_SUPPLEMENTAL

"The first Kuro5hin fundraising drive is officially over, and by any measure, it's been a fantastic success. Altogether you contributed over $37,000 in memberships, donations, and text ads, along with perhaps another $2500 in donation pledges, and an unknown amount in pay-by-mail memberships. I should be able to report a final 'grand total' in a few weeks."

Interesting. I need to get Reptile to a point where it is stable so that I can have a community of users who see value from the code base. As it currently stands Reptile is only usable by core developers. Hopefully this will change soon with a native OSX port

"Linking to or framing of any material on this site without the prior written consent of NPR is prohibited. If you would like to link to NPR from your Web site, please fill out the link permission request form."

Sorry... I forgot to request "permission" from you. If you have any problems feel free to contact me personally.

Wired also has the story .

"When huge, nameless, faceless corporations try to impose 'linking policies' upon webmasters who want to point to the company's site, people usually react in a predictable way. They get mad, they spitefully put up dozens of policy-violating links, and they bemoan, once more, the fact that some folks still don't understand that if you don't want to be linked you shouldn't be on the Web."

...

"'No matter how deep or shallow your link is, NPR requires you to fill in this form,' Doctorow wrote on the site, links included."