SSB Log Entry 1100

pigeon-protocol

Huge Brainstorming Doc - Part I

I am brainstorming ideas about better ways for Pigeon Protocol peers to attain consistency while preserving privacy, reducing meta-data and decreasing disk usage. I post this note publicly in hopes that others can share their feedback.

This turned into a 1,500 word essay. Sorry.

If you are well acquainted with SSB and Pigeon's design, you may skip straight to "How Does Pigeon Broadcast Intent Today?"

Before I get started, some background information.

How Pigeon references resources using formatted strings:

Pigeon, like SSB, uses specially formatted strings to represent resources:

?	SSB	Pigeon
A user/feed	@z2M8…eiHk=.ed25519	USER.09XB…0DGG
A message	%/DhQ…GWXk=.sha256	TEXT.RGKR…WHR0
A file/blob	&KzGj…bUJM=.sha256	FILE.0RHW…RKGR

SSB and Pigeon peers both keep a linear database of log entries.

SSB uses JSON to represent log entries:

{ "key": "%bf5/...23sk=.sha256", "value": { "previous": "%Q83+...25BU=.sha256", "author": "@6CAx...+Uv0=.ed25519", "sequence": 15, "timestamp": 14...16, "hash": "sha256", "content": { "type": "contact", "contact": "@p13z...E+vI=.ed25519", "following": true, "blocking": false }, "signature": "s2qZ...DQ==.sig.ed25519" }, "timestamp": 155...003 }

Unlike SSB, Pigeon uses a special text format for entries:

``` author USER.09XB…B0DGG depth 2 kind example lipmaa NONE prev TEXT.Z3QS1…CZGN0

file_name:FILE.622P…VGPG

signature W94B…100G ```

NOTE: Pigeon, like Bamboo supports a lipmaa property so that you can verify a feed without downloading the entire feed.

How Pigeon messages flow without a network:

A log message is useless if it only lives on your hard drive. You can't have a peer-to-peer technology without inter-peer data exchange.

SSB has a number of ways to broadcast and discover information such as via LAN or a network connection to a public pub server. The ultimate goal is to get data from one machine to another.

In Pigeon, when a peer wants to transfer data to a neighboring peer, it creates a specially formatted file directory on the local file system. This directory can be zipped (or not) and put on to a CD-R, FTP server, etc.. The protocol does not concern itself with transport level concerns and assumes that networks like the internet do not exist.

Here is an example of what a bundle looks like

Since the format of the bundle directory is part of the spec, a compliant Pigeon peer will be able to import another peer's bundle and attain eventual consistency with the rest of the network.

If your local node had three new blobs it wanted to broadcast, the exported bundle might look like this:

├── messages.pgn <= Important messages file! ├── 622PVGPG.blb <= Bunch of files or whatever ├── FV0FVMRG.blb └── YPF1PCSG.blb

In the example above, *.blb represents a single blob file. In this case, the peer had three new blobs that it wanted to spread across the network.

The real-world data transfer might happen as follows:

Bob sends Alice a DVD-R containing a few GBs worth of feed data.
Alice runs pigeon bundle consume /media/foo/bar/
Alice runs pigeon bundle create and burns the resulting output onto a DVD-R
Alice mails a DVD-R back to Bob
Bob runs pigeon bundle consume on the response data
The two peers have attained consistency.

Now let's talk about the messages.pgn file in there. Remember that example Pigeon message in the previous section? messages.pgn is a special file that contains a lot of these (example available here). It will contain the author's messages, plus messages for the author's friends. It's just a bunch of Pigeon TEXT. messages concatenated into a single file. Peers will add these messages to their local DB.

Broadcasting Intent and Needs - SSB

In SSB, you let your peers know who you are interested in (or want to block) by adding a message to your database.

Advantages:

Peer meta data lives in the same place as normal data.
Publicly stating your "stance" on a peer allows neighboring peers to transfer data more efficiently with less duplication.
You can calculate graphs of the network with this information
You can set your "hops" to fine tune how aggressively your node will assist peers in gaining consistency.

NOTE: A lot of these bullet points are based on conjecture. I am not an SSB dev, though I have spent time reading all the docs/whitepapers/etc..

Disadvantages

Meta-data is publicly and permanently broadcast, though there are partial workarounds.
A log is eternal. Every follow/unfollow/block adds "noise" to the log that does not necessarily help application developers.

How Does Pigeon Broadcast Intent Today?

I've never liked the way SSB uses the log to transmit protocol-specific (as opposed to application specific) information. With that being said, I can't think of too many alternatives and I may follow SSB's example if no better alternative is found.

When I wrote the bundle spec for the currently-in-development prototype, I intentionally avoided defining message schemas the way SSB does. My hope was that I would eventually find an alternative.

So how does Pigeon currently broadcast intent and needs without schemas for follows and blocks? Right now, it sort of doesn't. Bundles are just naive database dumps that broadcast everything to everyone every time. This works, but is horribly inefficient. Long term, we will need something better.

Now that we have set the stage, I can (finally) talk about my idea.