Reference
Everything HyperCrux offers, in one place: the Go calls, the commands, the SQL functions and the rules for keys, fields, links and vectors. The README on GitHub explains each of them at more length. FORMAT.md describes the file for programs that write it without HyperCrux, and EXPORT.md describes the export format.
The Rules
- Keys are written
table:id, such asdocs:7. The table name is lower-case letters, digits and underscores, starting with a letter, up to 63 characters. Anything can follow the colon, up to 1,024 bytes for the whole key. A key never changes. - Fields are the table’s columns. A put sets the fields it’s given, leaves the others alone, and adds a column for a field the table hasn’t seen. Setting a field to null clears it. Maps and lists are stored as JSON text.
- Vectors go in the field
vec, stored as float32 values. The first vector in a table sets the size for the whole table. A vector can’t be empty, all zeros, or contain NaN or infinity. - Links go one way, from one record to another, with a type of 1 to 200 characters, such as
ownsorcites. Both records must exist. Deleting a record deletes its links. - Distance is cosine distance: 0 for the same direction, 1 for unrelated, 2 for opposite. Search compares every vector that passes the filter, so results are exact.
Go
go get github.com/hypercrux/hypercrux
| Call | Does |
|---|---|
Open(path) |
Opens a file, creating it if needed |
Get(key) |
A record’s fields, or ErrNotFound |
Put(key, fields) |
Creates or updates a record |
Delete(key) |
Deletes a record and its links |
Scan(prefix, after, limit) |
Records whose keys start with prefix, in key order |
Exec, Query, QueryRow |
Plain SQL, with HyperCrux’s functions available |
Link(from, type, to) |
Links two records |
Unlink(from, type, to) |
Removes a link. An empty type removes every type |
Neighbours(key, dir, type) |
A record’s links: Out, In or Both |
Walk(key, dir, type, depth) |
Every record up to depth links away, nearest first |
Nearest(table, vector, k, where, args...) |
The k closest vectors, with an optional SQL filter |
Update(func(tx) error) |
Runs a function as one transaction. Tx has all the calls above |
Adopt(table) |
Makes a table created with plain SQL a record table, or brings one back in step after a schema change |
Drop(table) |
Deletes a record table with its records and their links |
Check() |
Confirms keys, rows, links and vectors agree |
Export(w) |
Writes every record table, record and link as JSON lines |
Import(r) |
Reads an export into a file without record tables, all in one transaction |
Errors that break a rule wrap ErrInvalid, and missing keys, links and tables wrap ErrNotFound. The full documentation is on pkg.go.dev.
The hypercrux Command
| Command | Does |
|---|---|
init FILE |
Creates an empty HyperCrux file, or adds HyperCrux’s tables to an SQLite database |
put FILE KEY [JSON] |
Stores fields from a JSON object, or from standard input |
get FILE KEY |
Prints a record as JSON |
delete FILE KEY |
Deletes a record and its links |
scan FILE PREFIX |
Lists records by key prefix. --after KEY, --limit N, --vec |
sql [--json] FILE STATEMENT [ARG...] |
Runs one SQL statement with ? arguments, taken as they are |
link FILE FROM TYPE TO |
Links two records |
unlink FILE FROM TYPE TO |
Removes a link. * as the type removes every type |
neighbours FILE KEY |
Lists a record’s links. --in, --both, --type T, --json |
walk FILE KEY DEPTH |
Lists records up to DEPTH links away. --in, --both, --type T, --json |
nearest FILE TABLE VECTOR |
Lists the closest vectors to a JSON array, or to a record’s own vector when given a key. -k N, --where SQL, --json |
adopt FILE TABLE |
Makes a plain SQL table a record table, or brings one back in step after a schema change |
drop FILE TABLE |
Deletes a record table with its records and their links |
check FILE |
Checks the whole file, and exits with status 1 if anything disagrees |
export FILE |
Writes every record table, record and link to standard output as JSON lines |
import FILE |
Reads an export from standard input into a new file, or one without record tables |
version |
Prints the version |
Only init, put and import create a file. The other commands refuse a file that doesn’t exist or isn’t a HyperCrux file. With sql, options go before FILE, and everything after FILE is the statement and its arguments.
SQL Functions
HyperCrux adds these to every connection it opens, in Go and in the hypercrux sql command.
| Function | Returns |
|---|---|
distance(a, b) |
The cosine distance between two vectors, each a stored blob or a JSON array such as '[0.1, 0.8]' |
vector(json) |
A JSON array of numbers as a stored vector |
walk(key, depth [, type [, direction]]) |
The keys within depth links, nearest first, as a JSON array. direction is out, in or both |
json_each(walk(...)) turns the walk into rows to join with, which is how one statement follows links, filters fields and ranks by similarity:
SELECT d.key, d.title
FROM json_each(walk('customer:42', 2)) w
JOIN docs d ON d.key = w.value
WHERE d.status = 'open' AND d.vec IS NOT NULL
ORDER BY distance(d.vec, '[1, 0, 0]')
LIMIT 10;
The File
A HyperCrux file is an SQLite database in WAL mode. Records live in ordinary tables. HyperCrux adds hc_meta, hc_tables, hc_keys and hc_links, and a few triggers that enforce the rules above for every program that inserts, updates or deletes rows. Schema changes are the exception: after dropping or rebuilding a record table with plain SQL, run drop or adopt. FORMAT.md has the details, including how to put, link and delete from other languages with plain SQL.
Export and Import
An export is JSON lines, one object a line: a first line naming the format, a line for each record table with its vector size and its fields in order, a line for each record, a line for each link, and a last line with the counts.
hypercrux export notes.db > notes.jsonl
hypercrux import copy.db < notes.jsonl
Integers, reals, text, bytes and vectors each keep their kind and their exact value, so exporting an imported file gives the same bytes. Export refuses a value that put wouldn’t take, such as an infinite number stored with plain SQL, and names the record. Import goes into a new file in one transaction, and refuses an export that was cut short. EXPORT.md has the details, for programs that read an export or write one to import.