my gists, just random but connected thoughts relating to things involving programming and distributed systems.
You can not select more than 25 topics Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long. 7.0KB

GoLib and GoScript

As GLib is to Vala shall Golib be to GoScript

1. GoLib

Think like GLib, the C-object library used by Gtk, Gnome, GIMP, etc. This is the standard library for GoScript. It implements all the automation and code generation functions as provided by GoScript.

What GoLib provides is not necessarily the most performant solution. For these cases, one can write a dangerous pointery thing directly in Go, embedded in GoScript embed blocks.


GoLib implements an abstract container format that roughly matches what is represented by JSON. Go has Maps for Objects and Slices for Arrays. They otherwise work roughly the same, and it is possible to dynamically generate a Gontainer tree.

The Gontainer is not extremely unweildy alone as written in Go, however, for the purposes of optimising cache locality, completely hand-written, dirty and even maybe C or assembler code for performance critical parts. The Gontainer is for general code, for complex, easy to tangle application core data, and reference implementations that may later be replaced with optimised components or wholly written in performance-grasping version.


Gontainers contain metadata relating to the repository location of the code that operates on them, including Go version, Gontainer version and GoScript version. The complete package also can be assembled by syncing from the repository and compositing it into the text format all of the code implementing as well as the state data.

State Journal

All operations create journal entries that encode each mutating operation as a transaction. Once the journal cache reaches its high wark, it compacts a snapshot and keeps a FIFO ring buffer of several recent snapshots. This is of course not trivial overhead so it is batched through a channel with a worker and passed through to the Badger database backend.

Badger handles the versioning history automatically, so the in memory state journal only keeps history until badger has flushed it. After this, it is possible to step back in the state history, the metadata of the journal is also kept in the database, which uses logical clocks to form a DAG.

In other words, all applications have an optional but easy to enable and customise undo history journal built in, and the application’s cache is the primary storage, but documents (branches of the Gontainer tree) can be exported into whatever format an exporter has been created for, and is necessary in order to transmit snapshots of data. If the tree is desired, this is already the base format used for text serializing of Gontainers.

2. GoScript

This is something like a Go-ified version of Clojure. It allows easy embedding of alien code, such as also Go, inside triple backticks ``` blocks prefixed by the name of the generator (eg go, enable the creation of new ones). It is not really a programming language but a script for generating Go code that is complete enough but strips everything down to the pure and functional.

As such, its allowed constructs adhere to the principles of Functional programming - no function mutates its caller’s data but instead mutates a copy and returns it. It is not difficult to write Go code that works this way, but it is wordy, and requires a lot of in-scope declared objects that are the framework. As such, in a conventional Go application, this means it would normally have to be copied into the package and package renamed. GoScript handles this automatically for its GoLib Gontainer system.

Hot Loading

As mentioned, it is possible to embed pure Go code into GoScript, and rather than burden the programmmer with extra complexity in the write/test/debug cycle, GoScript generates a boilerplate core server that runs an in-built Go compiler that is synced from the current latest version or as specified.

If the code is wholly contained in triple backtick escaped blocks it is constant and will be compiled once at compile time.

If the code in the block is generated at runtime (made of variable strings) for the time being this limits GoScript to amd64 platform which has the required plugin code. In theory even generated code can generate code.

Gontainers can be turned into JSON (or other) text format, and are the basic inter-Gontainer messaging format.

Ultra Simple Syntax

Derived from the design principle in Lisp-like languages, and Assembler languages, we have the Gontainer, built on Maps, and for order-sensitive and better cache locality, the Gist.

Gontainers are basically map syntax, except we only use string indices so we eliminate the quotes and use the suffix of : directly after the name string.

Gontainers have a “meta” field that contains info about the application, and underneath the “type” key are New, Init, Get, Put, Validate, Marshal, Unmarshal functions. The other item in type is the Gontainer specification, which sets things like type name, provides possible internal use of the code (the comment-stripped gofmted code that runs each of these functions can be presented to the user and processed by cold- and hot-loaded application code.

The Marshallers have the optional possibility of embedding the sourcecode, and the GoScript plugin execution engine can load such code, compile it and immediately start running it. When performance is critical, it makes more sense to rebuild the whole application tree and restart the application for updates. But GoScript allows you to do either, to build all as a monolithic module, or optionally isolate the lower-bandwidth using paths between components in hot-loadable components that can be separately stopped and restarted on new code without taking down the whole system.

GoScript constructs code on the basis of many overlapping and encapsulating closure scopes. Unlike Go, however, there is no global scope or package scope or whatever. GoScript has pure functions only, the first parameter is usually a context variable containing the data the function computes from, plus a body of parameters forming the Gontainer standard function parameterisation.

Unless a return type is specified on a GFunc, it always passes through its first parameter to the receiver. Because . means the same as / does in URLs, instead of arbitrary receiver names the scope locality is directly indicated by these dots.

Security Through Design

Instead of providing presumptive, prejudicial types of approaches to ensuring users of your library code don’t tinker with internals they shouldn’t, instead, there is one rule about who can see and change what inside a Gontainer tree: The nodes closer to the root see more than those closer to the leaves.

This is enforced literally through the inherent structural limitations of a center-outwards construction pattern for a directed acyclic graph. The root is the outermost container, and each layer inwards is a further layer of isolation. When two nodes in the graph need to interact with each other, there is a service inside the first layer of the container that brokers channels between nodes to transmit data.

Each Gontainer has a channel broker, and all Gontainers have an account with the root Gontainer Broker.