This introduction uses the `lk` cli with bash. You can follow along by downloading the latest release. If you're more comfortable with Python or want a more detailed explanation check out the guide.

lk --version

linkspace-cli linkspace-cli - 0.4.0 - main - 188b2ed - 1.74.0-nightly

Point

A single unit|event|message in linkspace is called a `point`. A point can hold just shy of 64kb.

echo "Hello, Sol!" | lk point > mylog

Points have a few of optional properties besides the data field.

a spacename:

echo -n some data | lk point ::/my/example/spacename --data-stdin >> mylog

a timestamp in microseconds since epoch using `now` by default:

STAMP=$(($(date +%s%N)/1000000)) # There's an easier syntax we'll get back to.
lk point ::/my/example/other_space --create-int $STAMP >> mylog

and a signature:

# We don't have a linkspace instance yet - we can still sign packets by creating/providing an Argon2 encrypted key
KEY=$(lk key --no-pubkey --no-lk --password 'my secret') 
lk point ::/my/example/spacename/subspace --sign --enckey "$KEY" --password 'my secret' >> mylog

All points are hashed using Blake3.

Point's refer to the hashed fields/data. The point, hash, and a header are packed into the packet format. Functions/APIs deal exclusively in the packet format.

`pktf` formats a stream of packets.

cat mylog | lk pktf "Hash: [hash:str] is a point at '[spacename:str]' with data '[data]'"

Hash: Yrs7iz3VznXh-ogv4aM62VmMNxXFiT4P24tIfVz9sTk is a point at '' with data 'Hello, Sol!
'
Hash: 1PqoPt6K8FREny_9lj38gWpxIFUNANVRUTcT9LC2EBk is a point at '/my/example/spacename' with data 'some data'
Hash: TBthALXT83GYp1e4KkEd5-zdbpd55-kQY02W8sOVvX0 is a point at '/my/example/other_space' with data ''
Hash: Q3mlWoS6PY-AcCWF8F12Rq21pZcTRzRAWUMgx5cHL50 is a point at '/my/example/spacename/subspace' with data ''

Because these properties are adjacent to 'data', they can be used for indexing and processing by general purpose tools like `lk filter`.

Here we take only the points with a spacename starting with `/my/example` and 1 additional component

cat mylog | lk filter ::/my/example:* | lk pktf "[hash:str] [spacename:str]"

1PqoPt6K8FREny_9lj38gWpxIFUNANVRUTcT9LC2EBk /my/example/spacename
TBthALXT83GYp1e4KkEd5-zdbpd55-kQY02W8sOVvX0 /my/example/other_space

Points are addressable by their hash. To reference one point from another you add a link. Each link is a 16 byte tag and 32 byte hash value. Tags can be anything. If less than 16 bytes are supplied it is left-padded with 0's.

HASH=($(cat mylog | lk pktf [hash:str]))
lk point "::/my/example/link" -- "my first link:${HASH[0]}" "other link:Yrs7iz3VznXh-ogv4aM62VmMNxXFiT4P24tIfVz9sTk" >> mylog

You can get fancy with `pktf` and `xargs`.

cat mylog | lk pktf "mytag:[hash:str]" | xargs lk point ::/my/example/link -- >> mylog

A tool like `lk collect` has a few additional options for building points linking to other points. However, the `lk` binary is meant to do simple things. Use the library and a better programming language when doing non-trivial stuff.

Creating packets with `lk point` detects what kind you're trying to build. But it is better to be explicit. The 3 types of points are: `datapoint`, `linkpoint`, or `keypoint` (a signed linkpoint).

NOTE: linkpoint and keypoint do not read data from stdin by default.

echo somedata | lk linkpoint ::/my/other_spacename --data-stdin | lk pktf "[hash:str] = [data]"

iAYhZcRAhxWjLC5prHradPIUWJzOOxR3cZCr7EUFN_A = somedata

Database

Linkspace is primarily its packet format. Using the database is optional.

The database and other tools exists to make it easier to build complex systems and packet flows.

export LK_DIR=/tmp/linkspace ; 
lk init ; 
cat mylog | lk save > /dev/null ; 

LkInfo { dir: "/tmp/linkspace" }

Instead of using `save` you can set a write destinations directly.

echo hello world | lk point --write db --write file:mylog --write stdout | lk pktf [data]

hello world

The database has three indices. A 'log', 'hash', and the 'tree' index. The log-index is ordered by receive order, the hash-index by the point hash, and the tree-index primarily by a point's [spacename, create stamp] (see the guide for full details).

The database is always accessed through the runtime. The runtime lets multiple processes/threads can read, write, and watch for new points.

The library API uses callbacks and a user-driven eventloop (guide). `lk` is focused on piping packets. Commands are `watch-log`, `watch-tree`, `watch-hash`. These are shorthand for `watch –mode ..`.

lk watch-tree ::/my:** | lk pktf "[spacename:str]" > ./watching &

[1] 13558

cat ./watching

/my/example/link
/my/example/link
/my/example/spacename
/my/example/other_space
/my/example/spacename/subspace

Adding a new point

lk linkpoint ::/my/my/my --write db

Wakes the watching threads to output the new point.

cat ./watching

/my/example/link
/my/example/link
/my/example/spacename
/my/example/other_space
/my/example/spacename/subspace
/my/my/my

Applications

There are two optional fields included in the hash not yet shown. The domain and group.

The domain is analogous to a IP port. An application pick a domain name (max 16 bytes). For example `imageboard`.

lk linkpoint imageboard:: | lk p "[domain:str]"

imageboard

Groups indicate the set of intended recipients. If a group exchange process is running, an application doesn't have to deal with sockets, (HTTP) endpoints, or other IO except for the user interface. The application can read, write, request from the group, and process packets using just the linkspace library.

Building an application is done by mapping an application state to and from linkspace packets (in the database). For example, a drawing application where multiple people can paint to a shared image board. A simple mapping could be:

• Images data are saved as data points
• Every link in a linkpoint is: a hash to an image, and a tag holding (x,y) coordinates.

Adding an image might look something like:

X=30 ; Y=200 ; IMG="https://upload.wikimedia.org/wikipedia/commons/3/35/Tux.svg" ; 
curl -s $IMG | lk datapoint > tux.pkt
IMG_HASH=$(cat tux.pkt | lk p "[hash:str]")
lk linkpoint imageboard:: -- $(printf "%08d%08d" "$X" "$Y"):$IMG_HASH >> tux.pkt
lk save --pkts ./tux.pkt # Instead of `cat` we can provide a file

Building an image requires the program to watch for new packets in `imageboard::`, and on every (new) point draw over the image.

lk watch-tree "imageboard::" --max 1 \
   | lk p "[hash:str] has the links:\n [links]" 

CB-zDzOubxM8J68QIGzw5F4YGvIF4jAzGkiNwAh8Fzc has the links:
 0000003000000200:Sz0ZZDWxKht-jbM7Tfkn0nis4tNoKNPH_kfI7JYUnY4

A link might reference a point that is not (yet) available on the device. An application has to decide how to handle the situation. In this example we'll just wait. Waiting can be done manually. e.g.

lk watch-tree "imageboard::" --max 1 \
   | lk p "[links]" \
   | cut -d':' -f2 \
   | xargs -i lk watch-hash "{}" \
   | lk pktf "got point [hash:str] which has [data_size:str] bytes"

got point Sz0ZZDWxKht-jbM7Tfkn0nis4tNoKNPH_kfI7JYUnY4 which has 49983 bytes

Or use `lk get-links`. It has a few common strategies.

lk watch-tree "imageboard::" --max 1 \
    | lk get-links pause \
    | lk pktf "[hash:str]"

Sz0ZZDWxKht-jbM7Tfkn0nis4tNoKNPH_kfI7JYUnY4
CB-zDzOubxM8J68QIGzw5F4YGvIF4jAzGkiNwAh8Fzc

To complete the imageboard application we'll have to add a few more steps to merge the data into a single picture. See the tutorial for an example on doing this and more.

The final piece of the puzzle is the group field. A group is 32 bytes to signal the intended set of recipients. It is orthogonal to the domain field, as the application should not care which group its running in.

PUB=$(echo "Hello, Sol!" | lk data | lk pktf "[hash:str]")
lk linkpoint :$PUB:/example | lk p "[group:str]"

Yrs7iz3VznXh-ogv4aM62VmMNxXFiT4P24tIfVz9sTk

If no group is specified (like we've been doing) the public group is used.

lk linkpoint :[#:pub]:/example  | lk p "[group:str]" 

Yrs7iz3VznXh-ogv4aM62VmMNxXFiT4P24tIfVz9sTk

The other special group is `[0;32]`, also called the private group. You can refer to it with the expression `[#:0]`. Functions/subcommands that read/write existing points skip and/or warn whenever a point from the private group is seen unless enabled with `–private`.

lk linkpoint domain:[#:0] | lk save 2>&1 # creating a packet is ok - but receving is not accepted by `lk save` without --private

error: Args { inner: ["/home/rs/Projects/linkspace/target/debug/lk", "save"] }
Pkt(
    PrivateGroup,
)

A system to exchange points in a group can be made from scratch. Linkspace does not prescribe a way to do so. Each group / network is different, and no single solution can cover every situation.

For example, use `lk watch imageboard:$MYGROUP | …` and forward the entire stream to another device using netcat/socat, ssh, email, http, a USB stick, or other way to exchange bytes.

Linkspace is designed to only ever be a streams of packets, without additional overhead of a (custom) serialization formats. As evident by the 'mylog' file we have used thus far. This keeps streams compatible with all tools that process streams.

To that end, each packet has a mutable header excluded from the hash.

Filters work on these mutable bytes as well. This let you quickly build specific network topologies.

netcat 10.0.0.1 -p 6000 | lk route ubits0:=:0000 | lk save & # get packets from a host and set their ubits0 to 0000
netcat 10.0.2.0 -p 6000 | lk route ubits0:=:0001 | lk save & # get packets from another host and set their ubits0 to 0001
lk linkpoint example::/hello | lk route ubits0:=:0002 | lk save # save my packets with ubits 0002
lk watch-log --asc example::/hello -- "ubits0:>:0000" | nc 10.0.0.1 -p 6000  & # forward all packets with ubits0 higher than 0000 back to host. 

A single linkspace instance can be used by multiple applications on device, and connect to others. To that end there are some conventions. These are functions that create/watch for point with some predefined spacename, links, and data format. Conventions enable interoperability between multiple applications and background processes.

One such convention is the `pull` convention. This writes a query as a specific point.

lk pull imageboard:: --write stdout | lk p "[spacename:str]\n\n[data]"

/pull/[b:Yrs7iz3VznXh-ogv4aM62VmMNxXFiT4P24tIfVz9sTk]/[a:imageboard]/default

:qid:default
type:1:[b2:00000010]
domain:=:[a:imageboard]
group:=:[b:Yrs7iz3VznXh-ogv4aM62VmMNxXFiT4P24tIfVz9sTk]
depth:=:[u8:0]

The goal of `pull` is to allow one process, e.g. an application like imageboard (bash) or mineweeper (python) to signal another process, e.g. a group exchange process like bash.exchange, that it wants packets matching a query from the group.

Queries define a 'set of points' in linkspace. The `filter` and `watch` commands are syntax sugar over queries. You can add `–print-query` to those commands to see the query used.

Queries are designed such that joining two query strings the result is the common subset of both or an error if the union is empty.

lk print-query example::/ok 

:mode:tree-desc
type:1:[b2:00000010]
domain:=:[a:example]
group:=:[b:Yrs7iz3VznXh-ogv4aM62VmMNxXFiT4P24tIfVz9sTk]
prefix:=:/ok
depth:=:[u8:1]

lk print-query example::/ok -- "spacename:=:/not_ok"

error: Args { inner: ["/home/rs/Projects/linkspace/target/debug/lk", "print-query", "example::/ok", "--", "spacename:=:/not_ok"] }
Error {
    context: "Error adding rule \'spacename\'",
    source: Error {
        context: "spacename:=:/not_ok",
        source: "space prefix conflict",
    },
}

That's it for this quick introduction. Some notes on high level algorithm design are worth a read. For a more in-depth technical guide or the library API see the Guide.