Kuro5hin.org: technology and culture, from the trenches
create account | help/FAQ | contact | links | search | IRC | site news
[ Everything | Diaries | Technology | Science | Culture | Politics | Media | News | Internet | Op-Ed | Fiction | Meta | MLP ]
We need your support: buy an ad | premium membership

HOWTO: Make a file sharing system.

By Surial in Media
Mon Jun 20, 2005 at 06:02:22 AM EST
Tags: Internet (all tags)

The RIAA and many other local facsimiles in nations outside the USA are aggressively persuing file sharers. More and more files on the file sharing services available are fakes; they do not contain what they portend to be, or contain only 5 to 7 seconds on a loop. Leechers reduce the average download speed considerably. Some manage to circumvent most of these problems but do not offer satisfactory search.

In short, there are plenty of reasons why it's time to introduce Yet Another File Sharing Service. This time, let's do it right.

Read on for a detailed description of how solve all the problems that current file sharing systems are struggling with.

Current Systems
Here's a short list of the mainstream abuse that occurs with current file sharing systems:
  • Leechers - people who upload far less than they download. In a P2P system, the fundamental rule is that each user 'should' approach a 1 on 1 relation between uploaded bytes and downloaded bytes. Any leechers need to be counteracted by people who upload more than they download.
  • Legal Threats - Just for uploading something it's possible (though for now unlikely) you'll get into legal trouble.
  • Fake Files - They come in many forms:
    • Media is only a few seconds, looped over and over
    • Media just has the wrong name.
    • Media quality is far worse than one would expect
    • Media contains advertisement for something.
    • Trojan horse hidden in the media (rare, as only certain formats can carry code and most media formats aren't among them).

The root cause for all these problems is a lack of accountability. If someone is responsible for putting up a fake file, either directly or indirectly (by not checking a file for correctness before putting it in the 'upload' folder), there are no repercussions.

Similar situation for leechers: Even systems that do accounting aren't enough; a leecher can just create a new identity on the fly.

So, let's solve all these problems in one fell swoop by adding accountability to the system.

Introducing: The social network.
You can draw your own social network quite easily: Put yourself in the middle. Add all your friends around you, in a circle, and connect them all to you using a line. Then, ask all of your friends to repeat this process by drawing their friends around their name, and letting their friends do the same, etc, etc, etc.

A social network is eminently accountable; if your friend is acting like a total ass, you can take measures. Aside from accosting them about their behaviour, you can simply drop them out of your circle of friends as a final alternative.

Using that axiom, we base our file sharing system on social networks. There's no central server, there's only a list of IP addresses of your friends.


Filtering out abusers
In order to 'log onto' this network, you first need a friend that's already on the network. If you don't know any, you can start a new little network with your friends. If you end up abusing the system by introducing any of the problems listed earlier, eventually you get kicked out by your friends. At some point you run out of friends (in theory, anyway), and you can't get back on the network.

Value trade
The program should offer a page listing all your friends and the trading 'balance' in various forms. There's the simple to measure 'uploaded versus downloaded', but also issues like: How often did I have to throw away a file because it was bad quality (assuming the downloading user can take the time to mark a file as 'bad', the software can assign penalty points), how often did this friend provide rare stuff I really wanted, and other esoteric criteria, limited only by how much time a user wants to spend notifying the software about how you rate the downloads from a friend.

The friend with the highest rating gets more of your upload allowance than other friends, ensuring that behaviour on the network you consider useful gets rewarded.

Dodging the RIAA
If you end up getting sued for uploading a file to a friend, that means your friend ratted you out. Your fault for adding that 'friend' to your list. Simple as that. Encryption can ensure that the RIAA massively inspecting network traffic won't endanger you legally.

Turning your circle of a friends into a whole network
If all you have is just your own friends, chances are you won't have access to a whole lot of material. You'll have to also access your friend's friend's resources, and your friend's friend's friend's resources, etcetera.

To accomplish this, you simply ask your friend to start looking for a certain resource on your behalf. Let's say Adam asks Ben to find some resource for him. Ben's friend Clara (who isn't a friend of Adam) has this file. In a perfect world, Ben would download the file from Clara, and Adam would download the file from Ben. However, because this wastes considerable bandwidth, for now we'll have to live with: Ben communicates Adam's IP address to Clara, and Clara can now elect to upload her file to Adam. She accounts this upload to Ben; Ben 'loses' standing with Clara. Similarly, Ben updates his status versus Clara positively and versus Adam negatively (Ben now 'owes' Clara but Adam 'owes' Ben).

At some point this has to stop; while you can trust your friends, and you can trust your friend's friends, at some point you really can't be sure anymore when you're dealing with leechers and the RIAA. The cutoff point is up to the user. Higher cutoff points, meaning there is more opportunity to upload, does mean you get better standings, which you can translate in having access to more files and faster downloads when you need something.

Logistics: How to make this a practical system
central-server-less designs have some problems. For one, you don't always know your friend's IP; not everyone even has a static IP. Furthermore, in order to 'do it right', as it were, it helps if you know you're -really- dealing with your friend and a spoofer. This requires encryption, and encryption without central servers requires PGP-like practices of handing out key hashes scribbled onto bits of paper to ensure your friend's public key is correct.

You can address both of these issues by offering a central logistics server (or using emule-like concepts where there are a number of third-party run servers) which takes care of being an impartial key verifier (if everyone trusts this server and everyone has the public key of this server, keys can be shared between any two persons by having them signed by the key authority) and distributing the IP numbers; all the software has to do is send a single 'ping' packet every so often to the server. Your identity name should be something easy to remember for everyone. Your email address seems like a prime candidate.

Compromise of the central server by ie the record labels is unlikely to be a disaster. Only if they also control local network trunks can they employ man-in-the-middle attacks, using the compromised key server to falsely claim their spoofed key is correct. This takes secret control of both the key server and substantial chunks of the internet network infrastructure. Right now, apparently, they can't even link IPs to a subscriber without a subpoena. Furthermore, the central server is fundamentally not doing anything illegal; they are not even distributing links to illegal content. They can market themselves as a general service that maps email addresses onto IP numbers and certified public keys. Such a server has plenty of legal use and as such would be hard to ban.

Another disadvantage of a decentralized network structure is that searches are potentially difficult. Concepts pioneered by existing file sharing services such as identifying files with a hashcode instead of a filename (as those are prone to change) should be used, of course. A search should work in concentric circles. You send a full search request packet which includes a randomly generated identifier to each of your friends. Only if all of those report back they don't have it for you, do you start asking each of your friends to start asking THEIR friends. All you have to do to accomplish this, is to send a hop count and the identifier to all your friends again; reasonably that won't amount to more than 160 bits at most, which is unlikely to cause a serious load on the network (searches did take up a decent chunk of network bandwidth on the gnutella file sharing system, also an attempt at a decentralized file sharing system).

Your friends showing up directly in your friends' network won't cause duplicate search results. If your friend unwittingly forwards your search request to one of your friends (who has already received this search request from you personally), your friend's computer can filter the duplicate search request easily; both have the same random identifier.

'value' is not just bytes uploaded versus bytes downloaded. One can imagine that someone 4 friends removed will not agree to download something to you unless he or she gets, let's say, triple credit for the effort, as a 'payback' against the increased risk of the target being a leecher or a RIAA informant. The downloader can then accept or reject this request. Each 'friend' in the chain in turn has to accept or reject; that's because the downloader doesn't directly get debited for the token value demanded by the uploader, as explained before. However, accountability always remains:

Let's say Clara, in our previous example, uploads this file to Adam. Adam's a leecher. Ben ends up being responsible for Adam's behaviour; Clara has debited the token value of her upload to Adam to Ben's account; if Ben doesn't return the favour somehow, he'll see dwindling efforts from Clara to supply him with good files at good download rates until, if Ben keeps vouching for leechers, Clara drops Ben from her circle of friends altogether.

Let's say Jack has a modem uplink, or crappy upload (some satellite internet services still have this), or pays per byte. He'd like to contribute to the network, but that costs him a lot of money. Let's say Jack has a friend, Kayleigh, who happens to be jacked in at 100MBit to the local university. Instead of paying through the nose at his ISP, he can of course give Kayleigh a bit of money for her troubles so that Kayleigh tells her software to artificially jack of Jack's rating. This way, Jack can leech all he likes from Kayleigh and her friends (and her friend's friends, etc, etc); Kayleigh will vouch for Jack, and with Kayleigh's luxurious connection, it's unlikely that Jack's leeching behaviour by itself is going to have a serious impact on Kayleigh's status as preferable 'file sharing friend'.

"Paying" for preferential access doesn't have to be done with real money. Someone who somehow manages to get excellent quality copies of movies far before they show up in theaters could get a lot of artificial tokens from friends even with a crappy connection. Someone who has a very large and tidy harddrive with everything named correctly, with subtitles and a high quality release, can also score bonus points (in the form of extra token value) from friends.

In other words, the system not only punishes bad behaviour, it also rewards good behaviour. That's something current systems definitely don't do; you don't get extra download speed on your other bittorrents if you're doing people the considerable service of seeding a very rare release, or some such. This proposed system does, and the accounting is timeless; you can upload something now, and reap the benefits a week later when you have a sudden craving to see some ancient movie or other that's no longer available in stores.

Just about all unwanted effects I've ever considered for a general file sharing service can be counteracted using token value and accountability. As such, it seems to me this system would be a vast improvement compared to the existing file sharing systems.

Naturally, existing cutting edge research on how to get the logistics right (such as the bittorrent protocol's various tricks) should be implemented here wherever possible. ie: Download from multiple sources, download random segments instead of always starting at the beginning, etcetera, etcetera.

Your turn!
I'm looking for comments. Am I missing something important? Has this idea been mentioned before? Is someone working on such a system? Might it be possible to retrofit an existing system with a social network accounting system?

Aside from looking for comments, I'm also trying to get someone to actually write it. I'm currently engaged elsewhere, already programming quite a lot for other projects. I only feel like so much programming per week, and unfortunately my other projects have priority.


Voxel dot net
o Managed Hosting
o VoxCAST Content Delivery
o Raw Infrastructure


Will it work?
o Heck yeah. Get to writing it! 14%
o Maybe. Write it and find out. 23%
o But internet users don't have any friends. 42%
o The RIAA and cohorts will find a way to shut it down. 19%

Votes: 21
Results | Other Polls

Related Links
o Also by Surial

Display: Sort:
HOWTO: Make a file sharing system. | 66 comments (48 topical, 18 editorial, 0 hidden)
Almost freenet (none / 0) (#2)
by StephenThompson on Fri Jun 17, 2005 at 11:04:18 PM EST

Freenet does much of what you describe, minus some of the security holes. In your scheme, if Adam aka RIAA gets Clara's IP address, he can supoena her harddrive which contains the illegal file. Freenet always routes encrypted data through several IP address and keeps files distributed so no single location contains illegal material. Thus, with Freenet, any IP address Adam aka RIAA got ahold of would not hold the files that he downloaded. Any freenet drive he subponeaed would only contain encrypted fragments which would not contain enough information to build a file. One of the big problems with freenet, and I think your scheme has as well, is slow file search.

Freenet? No. (none / 0) (#6)
by Surial on Fri Jun 17, 2005 at 11:48:56 PM EST

If someone is paranoid they can route file bits through the social network. This proposal doesnt take significantly more bandwidth per shared item, offers excellent search, and is still secure enough for day to day RIAA dodging. Again, if Adam is RIAA, and he manages to get a download from you, one of your friends ratted you out, basically.

Besides, no one uses freenet for run of the mill filesharing.
"is a signature" is a signature.

[ Parent ]

Note (none / 0) (#47)
by jolt rush soon on Mon Jun 20, 2005 at 08:17:42 AM EST

Freenet doesn't have a search.
Subosc — free electronic music.
[ Parent ]
Why... (2.50 / 6) (#4)
by BJH on Fri Jun 17, 2005 at 11:45:31 PM EST

...are you inciting people to take part in illegal activities?
Roses are red, violets are blue.
I'm schizophrenic, and so am I.
-- Oscar Levant

He legally owns all those files. (2.75 / 8) (#5)
by I HATE TROLLS on Fri Jun 17, 2005 at 11:46:51 PM EST

He just wants a backup.

[ Parent ]
Look at it from your perspective... (none / 1) (#11)
by Surial on Sat Jun 18, 2005 at 05:45:58 AM EST

How can you undermine a file sharing service based on social networks if you are the RIAA?
"is a signature" is a signature.

[ Parent ]
You tell one lie to one guy on the internet. (none / 1) (#21)
by I HATE TROLLS on Sat Jun 18, 2005 at 09:19:56 AM EST

Hard, isn't it?

[ Parent ]
Fuck off, douche bag (1.50 / 2) (#13)
by GreyGhost on Sat Jun 18, 2005 at 05:53:53 AM EST

Anything fun you might want to do online is probably going to be illegal.

[ Parent ]

Ah, logical arguments. (3.00 / 5) (#24)
by BJH on Sat Jun 18, 2005 at 01:12:40 PM EST

Or not.

Roses are red, violets are blue.
I'm schizophrenic, and so am I.
-- Oscar Levant

[ Parent ]
Where... (none / 0) (#50)
by reidbold on Mon Jun 20, 2005 at 11:30:03 AM EST

...does he incite people to take part in illegal activities? It looks like he just wants to build a P2P system, which is quite legitimate.

[ Parent ]
Riiight. (none / 0) (#59)
by BJH on Tue Jun 21, 2005 at 09:31:24 PM EST

And his comments about RIAA legal threats and informants and the dodging thereof are in there because he's worried about lawyers coming after his Linux distributions, eh?
Roses are red, violets are blue.
I'm schizophrenic, and so am I.
-- Oscar Levant

[ Parent ]
Not Everywhere (none / 0) (#63)
by cavegeek on Fri Jun 24, 2005 at 06:19:56 AM EST

Using P2Ps, and distributing copyrighted music and movies for free isn't illegal everywhere. It's not an inherently illegal activity.

[ Parent ]
My idea (none / 0) (#7)
by dhall on Fri Jun 17, 2005 at 11:53:37 PM EST

Construct a Kademlia-like* network. Bulk data are value-hashed and encrypted like the CHK's in freenet.

The golden rule is that uploading has a cost but downloading doesn't. Whenever you want someone else to upload something, you need to give them a reason why.

When you upload a big file, it gets scattered all over the network in smaller chunks. If you are uploading something really illegal, you would run your uploads through a proxy just to make sure.

When someone uploads something to you, you just say 'okay fine' - it's not like you have a choice whether to receive the packets, anyway.

When you download, you gather up the chunks. First you find out the hash of each chunk. Then for each hash, you look through your list of peers and pick the one closest to the hash. You ask him about the data. If he doesn't have it or doesn't want to give it to you, he tells you who you should try asking next. If he does have the data, then you have to give him a good reason why he should upload it to you.

The network is also built to carry dynamic things such as:
-Additive lists (eg. list of IPs) so you can jump into a bittorrent swarm or other standard P2P sharing.
-Mailboxes (writeable for those with public key)
-Dynamic pages (writeable for those with private key, like SSK in freenet)

* In Kademlia-like networks you have each node assigned a hash number. Nodes try to connect to other nodes with nearby hashes, and data are stored on nodes with hashes similar to the data's hash. The advantage of a Kademlia network is that if you know what you want, then you know where to find it.


I dunno, just throwing it out there.

Fails for the same reasons freenet will. (none / 0) (#14)
by Surial on Sat Jun 18, 2005 at 05:57:11 AM EST

I think that idea has merit, but the current harddrive capacity and network pipe of your average random internet user isn't big enough to support a network built around such principles.

Not yet anyway.

Right now you basically take what you can get - nodes are on- and off-line all the time, and you're lucky if even 1 person has what you're looking for. A system whereby only 1 server on the entire network has what you're looking for it extremely fragile. What if there's no good connection to this one guy?

Some questions for your plan:

 - how will search work?
 - how will you set up the network's IP lists in a manner that every Dick and Jane understands? All you need in my proposal is your friend's email addresses.
 - How do you entice someone to expend the network capacity to let you download something from them?

and the biggest one:

 - How do you secure the accounting for uploads? Unless there's some benevolent magic dictator machine that hands out unfakable credit for a good upload, how do you do it? The main story's system doesn't attempt to export credits beyond your direct circle of friends where people are assumed to honour each others credits out of a simple sense of friendship only.
"is a signature" is a signature.

[ Parent ]

yep (none / 0) (#23)
by dhall on Sat Jun 18, 2005 at 01:06:54 PM EST

I personally think that the search functionality needs to be built in a way that is scalable. What I do not want to see is something where you can flood the network with only a small bandwidth.

It could work like Emule's Kad search. Basically, you store the file info at the node that holds each of the the keywords for the file. Then when you search, it downloads the list for the first keyword, and filters. Or even better, find the file hashes on a website you trust, or from a friend. File hashes are small and easy to transport.

As for the node fragility, this is why I want to emphasize that the data store is not the most common mode of file sharing. The reason the data store is there is to provide a super-anonymous network for those that want it. There are ways to keep the fragility at bay: Upload to multiple nodes near the file, not just the nearest. And you can put in error correction like is done for file-splitting in Freenet.

For IP lists, all you need is one IP from the network and you're in. Once you connect, you first try to `find your place' by asking where your neighbors are.


Accountability is the most difficult question. Consider a system where any node uploads a signed public 'thank-you' as credit when someone else performs a request. Oops, you just broke anonymity, since you can look at who gave the thanks for a proxy.

This is why I was thinking that it would be good to put the data store and traditional direct filesharing together. When someone asks you to do something, you can say 'okay but only if you do [activity] for me'. Then, as the [activity] progresses, you perform your duties simultaneously. When one party stops, the other stops. If they say they won't do the deed, then you put them on the low priority queue.

So, if you are trying to publish something illegal, you would be already sharing some files that are less illegal. Whenever someone asks for one of those innocent files, you ask them to proxy something for you. Suppose you're A, the requester is B, and you want to anonymously upload to D.

Since B initiated the request, you still need a bit more anonymity. Let C be a random node.

If B wants you to give him a 1000 kB chunk, then you can transfer 500 kB to D via proxy. It works like this:

You ask B to perform 500 kB of proxy and 500 kB of duties for C. In the proxied communication with C, you ask him to perform 500 kB of proxy to D.

B has to upload 1000 kB, but he's getting the same for his trouble. C has to upload 500 kB, but again, he gets to request things from B.

This still leaves the question of what C wants B to do for him, but at least the problem is only half as big.

[ Parent ]

Help for non-leechers (none / 1) (#8)
by Kasreyn on Sat Jun 18, 2005 at 12:26:00 AM EST

If you run WinMX, which I do, get a copy of MXMonitor or LeecHammer. I use the latter, though many swear by the former. Basically, they're companion programs that automatically browse the shared collections of people downloading from you, and automatically cancel or block them if they're sharing fewer than x files or bytes, where x is a variable you choose. I've found, via LeecHammer, that almost exactly one third of WinMX users are zero-share leechers, so it's certainly a bandwidth savings to get rid of their asses. To anyone designing a distributed filesharing app, please take note and build leech-control features in. (I may just be behind the times here. My version of WinMX is probably outdated.)

Of course, you'll still have to deal with weirdos, just like anywhere online. For instance, people from foreign countries will message you incessantly in languages you don't speak, never seeming to catch on that you can't understand. Then there are the dumbasses who share their downloads directory so you wind up with mislabelled shit, and let's not forget the thousands of garbage files that are pissed into the file pool by our good friends at the media corporations. Other times you will encounter a breed of weirdo I call the "single extension asshole", who is interested in only ONE filetype - say, jpg's or avi's - and will cancel your download from his vast collection because your dozens of gigs of mp3's and pdf scans obviously mean you're a leech. There's nothing you can do about these except wish them a prompt, gruesome death and move on.

"Extenuating circumstance to be mentioned on Judgement Day:
We never asked to be born in the first place."

R.I.P. Kurt. You will be missed.
That really won't work. (none / 1) (#15)
by Surial on Sat Jun 18, 2005 at 06:00:55 AM EST

Once everyone starts using LeecHammer or other tools, you'll start seeing programs that generate random file lists by doing random searchers, and they never actually honour a request for download, or serve it very slowly. With random data. Now you've worsened the situation; searches are less reliable and there's a higher chance that the data you do manage to download is a fakefile.
If you add a system whereby you can hand out penalty points, what's preventing RIAA supporters from logging in, handing out random penalties all over the place, and generally playing havoc with the system?
There's nothing you can do about these except wish them a prompt, gruesome death and move on.
Uh, my proposed system solves it all.
"is a signature" is a signature.

[ Parent ]
Well, one of those isn't a problem (none / 0) (#18)
by Kasreyn on Sat Jun 18, 2005 at 06:43:20 AM EST

LeecHammer also can check to see if someone with $bignum files who has been on for $long time has any uploads going. If not, he's banned as a dubious character.

And as for generating random fake files, the solution to that is pretty simple. First, ban anyone who won't let you download from them as they download from you. Next, download a sampling of files from them and check if they're garbage. They can't be too large. No one's going to sacrifice gigabytes of their hard drive just to fool a leech detector. At least, not yet.

Right now, programs like LeecHammer and MXMon are not very widespread. I suppose things may change once the word gets out, but I wonder whether leechers will just shrug and share some files rather than going to the trouble of downloading and installing an anti-hammer garbage generator and sacrificing the necessary hard drive space. We'll just go over to measuring total bytes shared rather than total files shared, and the cost of successful leeching will be wasting tons of HD space. It won't be worth it to them.

"Extenuating circumstance to be mentioned on Judgement Day:
We never asked to be born in the first place."

R.I.P. Kurt. You will be missed.
[ Parent ]
Ease of use? (none / 0) (#28)
by Surial on Sat Jun 18, 2005 at 05:12:21 PM EST

Doesn't fix all problems. What if the guy with the fine collection of high quality files happends to be the RIAA checking out who they can add to the IP list to use in their next subpoena?

 Next, download a sampling of files from them and check if they're garbage. They can't be too large. No one's going to sacrifice gigabytes of their hard drive just to fool a leech detector. At least, not yet.

How do you garbage-detect? I can act like I have 20 gigs of music by faking it with only 100 mb of files, for example. It's too easy to screw around with any given system; without accountability somewhere the technical hurdles become sufficiently complicated (in my opinion), that whatever system you end up with is too inefficient or too complicated for Average Joe.

"is a signature" is a signature.

[ Parent ]

Distributed Networking. (none / 1) (#33)
by Kasreyn on Sat Jun 18, 2005 at 08:47:13 PM EST

What if the guy with the fine collection of high quality files happends to be the RIAA checking out who they can add to the IP list to use in their next subpoena?

That is the main reason I hate leechers. Everyone on a p2p network should be sharing the risk. And the principal risk is always for those who share a large amount of quality files - uploaders. The RIAA always chooses to attack people who are sharing dozens or hundreds of gigabytes of shit. Leechers, at least some of them, are afraid to upload at all.

The advantage of forcing everyone to share is that there will be too many of us for even the RIAA's deep pockets to prosecute. Yes, some will be busted. But it will remain a statistically low risk as long as wide-spead leeching can be prevented. Wide-spread leeching puts all the risk onto a very few quality uploaders, which is the real danger, because those uploaders may realize they're dangling in the wind with no protective camoflauge, and quit.

"Extenuating circumstance to be mentioned on Judgement Day:
We never asked to be born in the first place."

R.I.P. Kurt. You will be missed.
[ Parent ]
Good observation. (none / 1) (#36)
by Surial on Sun Jun 19, 2005 at 06:06:47 AM EST

Which calls into question the durability of these uploaders with amazing ratios.

Who are they? Why are they uploading so much? How long will they continue doing it?

"is a signature" is a signature.

[ Parent ]

what would you call me? (3.00 / 7) (#25)
by circletimessquare on Sat Jun 18, 2005 at 01:33:24 PM EST

my strategy, using emule:

load your shared folder up with gigs of porn. if you must download linkin park or evanescence, the kind of stuff the riaa is sniffing...

  1. stop all of your downloads except that song you want with the most sources and the best connections
  2. suck it down in under a minute
  3. immediately get it out of your shared folder
  4. if you do it fast enough, all the porn suckers you have cultivated will flood out anyone trying to get that drop of water pop song in your sea of masking porn
so what would you call me? a leecher?

i am in fact distributing material a lot of people want all of the time. on balance, i'm a supplier. but i am segregating types of material due to the legal environment i am working in. what would you do with someone like me?

however you slice it though, my strategy works: the riaa only cares about those who upload, so i'm pretty much completely safe, and i can get anything i want, and all i am screwing is the porn industry, which doesn't exactly keep me up at night, fuck them

The tigers of wrath are wiser than the horses of instruction.

A careful downloader? (3.00 / 3) (#29)
by Surial on Sat Jun 18, 2005 at 05:15:22 PM EST

It doesn't scale though. If everyone uploads porn, no one is uploading music.

Also sounds like a lot of trouble.
"is a signature" is a signature.

[ Parent ]

A proposal (3.00 / 4) (#37)
by zenador on Sun Jun 19, 2005 at 06:10:26 AM EST

Everyone in the USA follows circletimessquare's plan.
Those of us in other countries share the music you want and enjoy high speed porno downloads.

If only there was some way we could arrange this.

[ Parent ]

exactly (none / 1) (#39)
by circletimessquare on Sun Jun 19, 2005 at 11:39:00 AM EST

when i am downloading green day, i EXPECT that my copy is coming from dusseldorf or osaka

The tigers of wrath are wiser than the horses of instruction.

[ Parent ]
+1 Sp because... (3.00 / 2) (#31)
by fyngyrz on Sat Jun 18, 2005 at 07:02:29 PM EST

...I love watching these discussions.

Frankly, the only problem I have with ITunes and the like is I can't find everything I want. A lot of older stuff simply isn't there. But the ITunes model suits me just fine. I don't mind in any way, shape or form paying a buck for a good tune, and at the same time, being able to avoid the "filler" material that makes up the rest of many compilations. I get ten good tunes for the price of one CD, and that was certainly not the norm prior to the advent of purchasing singles online.

In fact, I heard a new Bruce Dickenson tune on XM today, hopped over to ITunes, and there it was. Now I own it. That's the way I like things. I like paying for good music, and not paying for music I wouldn't play on a bet. It puts a small force of my own on the music creation world, my own personal Darwinian improvement vector.

Blog, Photos.

Screw that. I buy legit CD's (none / 1) (#43)
by The Amazing Idiot on Mon Jun 20, 2005 at 12:30:44 AM EST

from 3'rd party dealers, like flea markets, used media stores, and cd/dvd used stores.

I like payiong 5$ for a movie and 2-3$ for a cd. Less time for hassle and better quality than I can wait for on BitTorrent.

[ Parent ]

The problem with iTunes... (none / 0) (#64)
by creature on Fri Jun 24, 2005 at 06:29:36 PM EST

... is that it doesn't stand up in the long run. Yeah, you'd pay a buck for a decent song. I would too, to be honest. But an iPod holds 10,000 songs. Are you going to spend 10 grand to fill it up?

[ Parent ]
eh? (none / 0) (#65)
by fyngyrz on Fri Jun 24, 2005 at 10:37:06 PM EST

I don't even own an IPod as yet, though my sweetheart has one and I'm perfectly ready to admit they're pretty nice. I just load up tunes into my computers and play them out to whatever I want to -- I have a bitching classic audio system and a to-drool-for home theatre system, and routing the music there is a matter of tapping a switch on a mixer. You get awesome management of playlists and so forth with the ITunes client, and frankly I like what it does better than any CD player or computer CD player client I've run into yet.

Next, I've got room for a lot more 10,000 songs, because I've got 800+ gigs of storage across my home network. So $10k doesn't begin to cover what I could spend, but then again, IPod or home network or shelves for CDs, who says I have to fill it up? Are you saying that because I have room for 10,000 CD's in my home, I have to spend (about) $150,000.00 on music??? Is there some cultural or moral imperative that says "storage must be filled" that I am not aware of?

Finally, not all music costs money. I'm a musician, and I can make music for free, sort of, if you don't count my time. It still has to be stored somewhere if I (or you) want to enjoy it later.

Remember: ITunes is not the IPod, and the IPod is not ITunes.

Blog, Photos.
[ Parent ]

+1SP (2.33 / 3) (#34)
by ccdotnet on Sun Jun 19, 2005 at 12:15:18 AM EST

Naive in the extreme - won't work at all. But +1SP because it's worth discussing.

Why? (none / 0) (#51)
by reidbold on Mon Jun 20, 2005 at 11:34:30 AM EST

[ Parent ]
Bittorrent is good enough (3.00 / 2) (#35)
by zenador on Sun Jun 19, 2005 at 03:14:11 AM EST

The fact that users can communicate via comments on the tracker sites mean that most of the problems in the article aren't even an issue. If the quality is bad or whatever people will complain and then everyone else will know, etc. Alot of trackers track how much you're sharing so you run the risk of being banned if you leech too much.

Um, it's good for legal stuff too.

Yup (none / 0) (#40)
by werner on Sun Jun 19, 2005 at 04:17:43 PM EST

But the anonimity just isn't there. The biggest flaw in the system, if you ask me.

[ Parent ]
Bittorrent is not good enough (none / 0) (#49)
by freestylefiend on Mon Jun 20, 2005 at 11:14:48 AM EST

I use BitTorrent. I don't need integrated search or chat, but I do need to be able to uniquely identify shared files to link to them. However, BitTorrent is not good enough, because transfers cease without the tracker. I wish that the eDonkey or Kademila networks had captured the niche that BitTorrent fills, but perhaps trackerless BitTorrent will do the trick.

MUTE looks promising for anonymity, but it doesn't support anything like Magnet or ED2K links and it is unsuitable for unregulated and permitted material because of its lower performance.

[ Parent ]

Poor design, but worth discussing (3.00 / 2) (#44)
by jd on Mon Jun 20, 2005 at 02:33:07 AM EST

File-sharing doesn't have to be "peer-to-peer", it can use multicast streams which mean that the source is the global address.

Secondly, if you are using indirect peer-to-peer, (ie: a grid layout, with all machines acting as proxy/cache systems), with all connections encrypted end-to-end, mid-point machines won't know what they are carrying and so their users can't "rat" on anything.

Thirdly, fake data can be easily identified. You group all files into blocks, and generate a SHA-1 and Whirlpool hash for each. You generate a similar hash for the whole file. Files that have identical hashes along their length are bogus, as are files were blocks do not match their hash values. (By using two radically different hash systems, it is extremely unlikely both could be broken. MD5 and SHA-1 are too similar, in this regard, and should not be used this way.)

Fourth, there are many P2P systems out there - Chord, eDonkey/Overnet, Gnutella, virtually any Grid package could be used for this purpose - the odds of having to reinvent the wheel are extremely slim. Most likely, you've got to figure out what the wheel you want looks like, then get one of those.

file sharing is good will not acquaintances (3.00 / 2) (#45)
by dimaq on Mon Jun 20, 2005 at 04:01:28 AM EST

that's my biggest argument against what you're proposing - ages ago, before donkeys and torrents there were friend ftps and even fxps for the real traders (never used). Trust worked well, selection worked well (you could ask what was good), but coverage sucked - contemporary file sharing mechanism gave much wider exposure to your files and more importantly to those willing to rip and distribute without really getting much back - that's good will, no friendship needed.

Here's a legal observation... (none / 1) (#46)
by Surial on Mon Jun 20, 2005 at 06:30:43 AM EST

If this network evolves to a state where the general maximum hop count is very low (let's say, 2), you're basically back to how it all used to work a long time ago. You borrow music from your friends. It's not exactly equal to fair use I guess, but, it's at least a lot closer to that concept than emule/winmx/etc.

"is a signature" is a signature.

Critical Mass (none / 0) (#48)
by jolt rush soon on Mon Jun 20, 2005 at 08:50:00 AM EST

The main practical problem is how you get this system started. Not only would I have to download and install another p2p filesharing program, I'll have to get all my friends to install it to. Only when they've got their friends to install it will I be able to get any new content.
Subosc — free electronic music.
Gnunet's pretty much like this (none / 1) (#52)
by m50d on Mon Jun 20, 2005 at 12:28:57 PM EST

A few differences, which mostly make sense:
You can request a random peer from web caches. This is a good thing from the perspective of bootstrapping the network, if you have to know a friend on the network to get on the network it can't grow very well. There would be people offering to be friends over the web anyway. You can turn off getting nodes off the web to make it purely friend to friend.
No central keyserver. The problem you miss is that if the RIAA simply takes the keyserver offline (which they probably can, you can't afford to fight it) that's enough to make the whole key system fail to work. Geeky friends (who would be the only ones interested in a new filesharing network, at least at the start) know each other's IPs already. And if you're real friends you can phone each other and say your IP, or something.
No advanced economics. It's just upload/download, with junk bytes (i.e. not matching the hash) penalised. There are a few things to stop junk content corrupting things though, mainly the ability to publish signed content under a psuedonym. If you know "bob500" is a source for reliable movie rips, you can search for files signed by him, and it's not fakeable. And of course this will work through the simple economics to mean that people get rewarded for sharing the good files (because more people download them)
Give it a look. www.gnunet.org

This kind of project has already been started (none / 0) (#53)
by CjDj on Mon Jun 20, 2005 at 10:57:05 PM EST

Although it might be hard to find, and nothing has been released yet, I started a project called vitello on sourceforge a while back that does (or will do) almost exactly what you are trying to accomplish, except maybe for the 'value' stuff.

There is a notes.txt file that is in the sourceforge CVS that gives a general description of what the project is designed to do.

Some coding was done, although the code has not been checked in for some time, so the CVS is rather out-dated.

Not much work has been done on it recently, but I intended to continue working on it soon.

Samizdat protocol (none / 0) (#55)
by kiwipeso on Tue Jun 21, 2005 at 04:53:05 AM EST

How about not doing it socially and instead doing it as an encrypted network with a proxy? I'm doing one slowly that way.
I would say that all you need is to get people with fast enough connections to become proxy servers and then freely exchange the keys by approved network links.
I have figured out a way to spread out in chunks the files being exchanged with a good encryption system. What you really need to do is just vote on the reliability of the proxy servers and then it will route around censorship or other network damage.

There is an old writeup at http://everything2.com/index.pl?node=Samizdat%20protocol
Kaos operating system creator.

Nice concept... many many problems (none / 0) (#56)
by aapiero on Tue Jun 21, 2005 at 12:04:32 PM EST

Hi there,

we, at work, already discussed about the same idea some weeks ago and we were very excited about it but after a deeper thinking we thought it was too hard to realize and difficult to use.
Here are some open points:
  • What about users behind a NAT?
  • Where can you make money? ;-)
  • What about performance?
Probably you'll end with a nice niche system like pgp (not hard for me or you but impossible to understand for the classical Joe user).
After saying this I'm totally open to suggestion and discussion on the problem... until now I did not read any constructive post ;-)


Those are addressable (none / 0) (#57)
by Surial on Tue Jun 21, 2005 at 12:47:41 PM EST

The NAT users you serve by letting the central key server and IP broker notify the non-NAT party to connect to the NAT party. 2 NATters can't connect to each other. This is annoying, but no more of a restriction than every other file sharing algorithm out there that doesn't rely on a bunch of megapipes from universities or some such.

Money making? No where. Good luck making money selling software oriented to users, and especially good luck selling software that is also useful to a large group of folks who hack software for a hobby. So, there is no money.

Performance should be far better than any other attempt at social networking because unlike ie freenet and such it does NOT send the same bytes through multiple routings. Also, unlike other networks, leechers should be almost non-existant, raising the average download speed considerably.

As far as the PGP flop - yes, I thought of that, which is why there's a central server in the first place. All you do is copy your address book into the application and the rest is taken care of - exactly analogous to how you set up AIM, ICQ, MSN, etcetera, and many many millions of average joes can manage that one.

"is a signature" is a signature.

[ Parent ]

Central server leads to napster failure (none / 0) (#60)
by aapiero on Wed Jun 22, 2005 at 04:55:30 AM EST

[...]PGP flop - yes, I thought of that, which is why there's a central server in the first place. All you do is [...]

The central server approach is what you really like to avoid after the napster case...
Without central server (too easy to be put offline) you remain with a bunch of new problems:
  • How to exchange public keys for cryptography?
  • How to know when a friend is on-line?
  • How to know where to send/get bytes?
I repeat myself... the idea is quite good but very hard to realize it in the right manner.


[ Parent ]
In fact... (none / 0) (#58)
by Surial on Tue Jun 21, 2005 at 12:51:37 PM EST

as MSN is also based on emails it shouldn't be too difficult to write a system that offers to copy the 'addressbook' from your MSN client. I think Aimster or what's it called (file sharing to your AIM contacts) gained a (brief?) period of success. This is similar except it offers better automated accountability and the option to extend to the second circle or even beyond that means you get access to far more files, if at reduced download speeds.
"is a signature" is a signature.

[ Parent ]
This sounds like the K5-User-Sponsorship problem. (none / 0) (#61)
by Imperfect on Wed Jun 22, 2005 at 04:17:21 PM EST

If I want on the network, I have to have a friend on the network. So far, so good. Unfortunately, this is easy to game. Someone, or likely many someones will start sponsoring anyone and everyone. Ban them? Okay, fine. But does this ban everyone they sponsored? If not, then you likely have alter-aliases of that person in the "extended" group. If so, then you cut off and alienate a sizable chunk of valid users who simply won't come back to your network. And for every user who doesn't come back to the network, you halve the value of that network. All of a sudden you have a very unfriendly and unpopular network that never reaches critical mass and fails -- not spectaculary -- but rather silently. It may not even be lamented on the front page of Slashdot. Other than that, a nice try.

Not perfect, not quite.
This has already been done (none / 0) (#62)
by konichiwa on Thu Jun 23, 2005 at 04:30:44 PM EST

...and it didn't turn out so well. A lot of the recent news about the closing of BitTorrent sites all over the internet showed that this idea is good and would require pefect implementation to work. TorrentBytes, elitetorrents, etc etc ad nauseum are (were) all sites that had most of the elements you're advocating as part of them. Not necessarily the social network part, but more of a ebay or epinions-style where as you uploaded more (relative to your downloading, of course), you were moved up the "ranks" and were able to download more files and faster. It didn't stop the RIAA/MPAA/BSA/etc

Maybe it already exists. (none / 0) (#66)
by 3vi1 on Sat Jun 25, 2005 at 10:32:43 PM EST

You forgot the part where you don't publicize that the network even exists nor make the client available outside your social network.  It might already exist, for all we know.

"Furthermore, the central server is fundamentally not doing anything illegal".  Web sites have been sued just for linking to other sites then link to copywritten works, which is about the same level of enablement.  Do you really think they couldn't find a single judge to produce a search warrant, and that they wouldn't also find a ton of illegal MP3's/Movies when they raid the guy running this server? (Since you made it clear that the purpose of your network is solely to distribute illegal content).

Seriously though:  If you're the author of a program that goes to these kinds of extremes to make sure that illegal files can be traded efficiently, you're just setting the RIAA up for a legal victory, or many legal victories if they use the e-mail addresses to raid all 200 people on the network and offer them $10k settlements.

HOWTO: Make a file sharing system. | 66 comments (48 topical, 18 editorial, 0 hidden)
Display: Sort:


All trademarks and copyrights on this page are owned by their respective companies. The Rest 2000 - Present Kuro5hin.org Inc.
See our legalese page for copyright policies. Please also read our Privacy Policy.
Kuro5hin.org is powered by Free Software, including Apache, Perl, and Linux, The Scoop Engine that runs this site is freely available, under the terms of the GPL.
Need some help? Email help@kuro5hin.org.
My heart's the long stairs.

Powered by Scoop create account | help/FAQ | mission | links | search | IRC | YOU choose the stories!