a general purpose supernet

Basics

The internet of streams

The digital world you know is built on structuring streams of data. At its core the internet¹ attempts to provide a model where:

For any two connected devices running any application, there exists a connection to transmit data.

To do this it uses the following types of packets to transmit data between two devices.

Packets are transmitted and - using the IP address - they reach their destination: a device such as the one you’re using right now. On the device an application is running that looks for packets with a specific port to arrive.

In transit, a packet can get lost or corrupted. The result is that packets don’t arrive in the order they were sent. By adding a sequence number, the destination can verify all packets have arrived and reorder them to the order they were sent in.

I imagine the proto-internet was first discovered when people realized that packet SEQUENCE IDs are essential complexity. When transmitting data, something must define the order of packets. Consequently the physical route each packet takes is irrelevant.

The result is that conceptually each application on each device can talk to any other application on any other device.

This model is ideal for phone-calls or video streams. To build more interesting applications we create protocols to add further structure to the stream of data. There are thousands of different protocols, but what most of them have in common is a way to transmit questions and answers.

A couple of well known internet protocols that have this property are:

System	Question	Answer
DNS	archive.org	207.241.224.2
HTTP	/forum/index.html	Hello world!
FTP	/Projects/linkspace/readme.md	In a supernet […]
SQL	SELECT * from MSG where ID=1;	A message in a db

Using these tools we do not just create websites to provide information. At first we used these tools to create websites to provide information. But this has changed and we instead build ‘platforms’. Services that store, forward, and administrate messages so multiple that multiple devices can communicate.

We have reached the limit of using this model of streaming questions/answers paradigm for multi-party communication.

First, the stream abstraction can leak. Streams get disconnected, the other side hangs up, the other side is overloaded, etc.

Then, when more than two devices are involved the leaks compound. Synchronization is as slow as the slowest link, answers become stale or wrong.

Alternatively you design a system as a supernet.

Once you start thinking in terms of supernets, it becomes clear that many problems with streaming questions/answers are accidental complexity. When a supernet is used to build an application those problems become irrelevant or trivial² - and we can do things that are practically impossible if we keep talking streams.

Linkspace

Linkspace attempts to provide a model where: for any group running any application, there exists a space to address data³.

If the current internet is essentially streams for key-value systems, so you can talk to server, then linkspace is essentially a shared key-value space, so groups can talk about data.

A unit in linkspace is called a point. Each point has data, some auxiliary fields, and is uniquely identified by its hash.

To understand what each field does lets start with a simple example of a message forum.

The “image/BrokenMachine.jpg” is called a path and maps to [image data]. So far this should look familiar as it is similar to files in directories. I’ll refer to each entry as a point, and multiple entries as a set. The example shown has two sets merging. The result is a new set with 3 messages.

One of the most useful aspects of linkspace is a way to talk and think about digital communication in terms of a set of points instead of connections.

Practically any digital communication can be understood as merging sets of points.

Online platforms have dubbed different words for actions you can take: ‘creating posts’, ‘uploading image’, ‘upvote/like a post’, ‘stream a video’, etc. Fundamentally they can be understood as a frontend application providing an interface to merge sets of points. Either on your device or on theirs⁴.

The internet we use today has a single host design. For instance, a web-browser or app contacts http://www.some_platform.com for the key /image/BrokenMachine.jpg to get an image.

This is simple, but it has downsides.

There is a misconception on what an address is⁵, a host can get disconnected, you can’t (re)share and (re)use your copy of the data, and there is no standard on what happens when two people create two different image/BrokenMachine.jpg but with different pictures.

I would argue these fall under accidental complexity.

Especially the last one. Once the speed of light is measurable in a network, it requires a specific design to avoid two or more computers to write to the same path.

In our single host design, the data is hosted on a server and the person who has administrative access to that server can then administrate which one is the ‘real’ copy, and which one should be forgotten.

In linkspace there is no such thing as a ‘real’ copy on a single host.

Every path can refer to multiple points.

Each point is hashed. i.e. there exists a unique 32 bytes (or ~77-digit number) that uniquely identifies the point⁶ (which I’ll show as [HASH_0] instead of typing out).

It doesn’t matter when or where sets are merged - the result only has a single copy per message.

We can uniquely get a specific point by its [HASH_0], or multiple entries through a path “/thread/Tabs or spaces/msg”.

This might seem more trouble than existing solutions like a filesystem or HTTP. In those, one request by name gets you a single result.

However, this is not a real issue for two reasons.

In practice it is trivial to only request ‘the latest value’ or ‘the latest value signed by someone you trust’.

The later is especially interesting. If an application only requests points signed by a specific key, it effectivly administrates similar to how it is done in our current single host-administrator design. However, it has the additional property that it can be independent from hosting the data. i.e. in linkspace ‘hosting data’ and ‘content administration’ can be decoupled.

A point has two preceding fields. A group that signal who can read/write to the tree, and a domain field to indicate which application should read it. Essentially any pair of (domain, group) has its own tree.

For example the msg_board application and the [#:example] group.

Note that the group and public keys are actually 32 bytes. For brevity I used the LNS representation for [@:alice:salesexample] and [#:example]. LNS solves a similar problem as DNS, i.e. turning a name like www.example.com into a number like 192.168.0.1.

Finally, the messages we used as an example have a [HASH] directly in their data field. This would not work well for most use-cases. Instead, a point in linkspace has a list of links adjacent to the data.

For the full specification of creating and writing points see the guide

There are some nuances and various advanced topics. However, this should be enough to reason about the basics:

Users generate their identity, together they form groups and set up a method to exchange data.

The result is that conceptually an application only needs to process the state of the trees.

Ready to give it a try?

Linkspace is not an end-user application. It is a software library and command line tools. A GUI frontend to manage groups/domains/keys is outside its scope.

The packet format is stable. Points created will stay readable in future versions. The API is mostly stable but will have some breaking changes and additional conventions.

To give it a try you can Download the pre-build CLI and python bindings to follow along with the tutorial or the more technical Guide,

For the adventurous there is initial support for wasm, and a POC HTTP bridge called Webbit that works similar to WebDAV.

Q&A

Who is linkspace for?

Me from 10 years ago, me today, the next generation that wants their internet to be better, and everyone else. In that order.

The tool

If you’re just interested in the tool; linkspace can shrink a ‘stack’ significantly.

Data formats change and grow as projects goes on. Using linkspace to carry the data many potentially-critical features are built in at the basics. Instead of risking a costly and complex upgrade when you need a feature yesterday, you get a lot of properties by default such as: uniquely addressable/cross referencing events, security against tampering, logging, secure user identities, low effort backups, etc.

It is scaleable decoupled data creation, routing, storing, and processing; with data authenticity and user authentication to control access, wire up business logic, or log ‘who’ did ‘what’ and ‘when’. Take the parts you want and leave the rest out.

It even runs in the browser⁸, meaning you can reuse more code logic and tools.

Finally, by talking about data there is a lot of potential for a ‘network effect’. e.g. a support-ticket references the exact IoT event, a sale can references the your exact history of the object, etc.

The idea

I build linkspace so I (and hopefully others) can:

• define, prototype, build, and run real-serverless offline-first apps quickly.
• build an application without having to decide how to administrate for everyone that uses it - that responsibility sits with the (group of) users.

I hope it might solve some of the worst problems with the current internet.

Perhaps the greatest insanity of this time (and I believe its defining feature) is a few for-profit advertisement / propaganda services are the host-administrators of the digital ‘public squares’ for more than 2³² people alive.

It is questionable for people (and the groups they organize in) to be subjected to the interests of an unaccountable administrator. But I think the real impact is the medium as the message: You are not in control, surrender your minds for profit and submit to apathy.

It is a staggering amount of wasted potential for people to not experience and learn to building something for, and with, a group outside the limits and interest of a higher-authority-incorporated.

We are all lesser for it.

The message of linkspace should be: You are in control, and you should build the best places you can.

Is linkspace a blockchain?

No.

Blockchains and supernets share a common idea:

Using cryptographic hashes and public keys to provide a model for communication data that transcends the connection used to share it.

Blockchain use the cryptography to batch multiple events into ‘blocks’. As new blocks are made they are linked or ‘chained’ to the previous block. This creates a consensus of all events that happened.

A blockchain’s goal is to provide a model and tools to make consensus simple.

However, consensus on all events isn’t that useful as a foundational feature for the vast majority of applications.

Supernets don’t bother with a global truth. Their goal is to work with the links between packets. Consequently, it’s not difficult to define a ‘blockchain’ style consensus in a general purpose supernet.

Won’t an app have a lot of overhead compared to a basic Web server?

If all you want to do is stream one movie from a single host and forget it, then linkspace might be too much overhead. Few projects stay that simple. Most projects grow in scope: to identify users, save their comments, add them to groups, scale beyond a single server, etc.

Once a full stack is build, linkspace is very small w.r.t. its features.

As far as overhead goes, it is designed to be fast/low energy such that a low-end phone can use it to stream video.

Can you ask people to deal with the added complexity?

Yes.

Linkspace lack 6 decades of tooling that made the internet and web relatively easy for users, but that can change.

But is it worth it?

Yes.

Technically supernets better model the reality of multi party communication - asynchronous and authenticated⁹ In the long run they could end up with less moving parts and with fewer configurations.

For society it is important that anyone can take responsibility if they want.

Isn’t it a good thing that a host administrates what I and others see online?

Sure.

There are different ways to do so. The straight forward approach is to have the application or user trust the public key of someone to scan and whitelist content. This emulates the current system of ‘admins’, while still having users give the option to replace them.

Alternatively can be more dynamic. You could have friends or friends-of-friends vouch for content.

I suspect the latter to become more important as AI drives the cost of bullshit to zero and while platforms have no incentive to identify the engagement and content from chatbots.

Won’t we end up with the same paradigm of highly centralized control?

Maybe, maybe not. If user today could walk away from a host-administrator without losing their history, identity, and links to others; then they would get a better deal then they do now.

Why not [alternative]?

There are two similar types of systems that can’t do what I want.

• Synchronizing chain of trust: Is slow and not useful for the vast majority of users - (and easy to emulate in a supernet).
• Faster email: It’s not that fast and it is using server-defined authenticity - plus most of the list below.

Other supernet-like systems are limited in some way or simply choose a different design:

• Too specialized. E.g. Git has a lot of plumbing for diffing text.
• Large ‘packets’ - a hash might refer to gigabytes. This requires multiple levels to deal with fragmentation in multiple ways.
• Either its primarily hashes, or its primarily path names. Linkspace has both as first class addresses.
• Too slow. Packet routing/parsing should be doable in just a few instructions - ideally possible in an integrated circuit. It should be fast enough to stream video without using a second protocol. That means no json or base64.
• No Groups. Setting who you share with and how is not supported or its little more than ‘run multiple instances’.
• No domains. Everything becomes one app with premature-bureaucracy and an ever expanding set of compatibility requirements that grinds development/experiments to a halt.
• Its focused on signatures and consensus.
• Poor scripting support.
• Excessively interwoven components. e.g. Transmitting packets require a fully running ‘node’ with a fixed method of exchanging or saving data.

That does not mean I think alternatives are necessarily worse. Different systems have different strong points.