A lightweight, embedded, LSM-tree database for Elixir.
Goblin is a persistent key-value store with ACID transactions, crash recovery, and automatic background compaction. It runs inside your application's supervision tree.
Starting a database
{:ok, db} = Goblin.start_link(
name: MyApp.DB,
data_dir: "/path/to/db"
)Basic operations
Goblin.put(db, :alice, "Alice")
Goblin.get(db, :alice)
# => "Alice"
Goblin.remove(db, :alice)
Goblin.get(db, :alice)
# => nilBatch operations
Goblin.put_multi(db, [{:alice, "Alice"}, {:bob, "Bob"}])
Goblin.get_multi(db, [:alice, :bob])
# => [{:alice, "Alice"}, {:bob, "Bob"}]Transactions
Goblin.transaction(db, fn tx ->
counter = Goblin.Tx.get(tx, :counter, default: 0)
tx
|> Goblin.Tx.put(:counter, counter + 1)
|> Goblin.Tx.commit()
end)
# => :okSee start_link/1 for configuration options.
Summary
Functions
Compare and swap a key.
Returns whether background compaction is currently running.
Exports a snapshot of the database as a .tar.gz archive.
Returns whether a memory-to-disk flush is currently running.
Retrieves the value associated with a key.
Updates a key and returns a value.
Updates multiple keys and returns a value.
Retrieves values for multiple keys in a single read.
Checks whether a key is a member of the database or not.
Writes a key-value pair to the database.
Writes multiple key-value pairs in a single transaction.
Performs a read-only transaction.
Removes a key from the database.
Removes multiple keys from the database in a single transaction.
Returns a stream of key-value pairs, optionally bounded by a range. Captures snapshots at enumeration.
Starts the database, see start_link/1 for more details.
Starts the database.
Stops the database.
Executes a read-write transaction.
Updates a key.
Updates multiple keys.
Functions
Compare and swap a key.
Compare and swap a value corresponding to key.
Returns true if swapped, false otherwise.
Comparison is done via ==.
Parameters
db- The database server (PID or registered name)key- The key to updateold- The value to compare withnew- The value to swap toopts- A keyword list with the following options (default:[])::tag- Tag to namespace the keys under:timeout- Timeout (in milliseconds) for the calls (default::infinity)
Returns
trueif swappedfalseif not swapped
Examples
Goblin.cas(db, :alice, "alice", "ALICE")
# => true
@spec child_spec(keyword()) :: Supervisor.child_spec()
@spec compacting?( :gen_statem.server_ref(), keyword() ) :: boolean()
Returns whether background compaction is currently running.
@spec export(:gen_statem.server_ref(), Path.t(), keyword()) :: {:ok, Path.t()} | {:error, term()}
Exports a snapshot of the database as a .tar.gz archive.
The archive can be unpacked and used as the data_dir for a new
database instance, acting as a backup.
The export is run inside the server, thus blocking file deletion and writes until completed.
Parameters
db- The database server (PID or registered name)export_dir- Directory to place the exported.tar.gzfile
Returns
{:ok, export_path}- Path to the created archive{:error, reason}- If an error occurred
Examples
Goblin.export(db, "/backups")
# => {:ok, "/backups/goblin_20260220T120000Z.tar.gz"}
@spec flushing?( :gen_statem.server_ref(), keyword() ) :: boolean()
Returns whether a memory-to-disk flush is currently running.
Retrieves the value associated with a key.
Returns the default value if the key is not found.
Raises Goblin.IOError if the underlying storage cannot be read.
Parameters
db- The database server (PID or registered name)key- The key to look upopts- A keyword list with the following options (default:[])::tag- Tag the key is namespaced under:default- Value to return ifkeyis not found (default:nil)
Returns
- The value associated with the key, or
defaultif not found
Examples
Goblin.get(db, :alice)
# => "Alice"
Goblin.get(db, :nonexistent)
# => nil
Goblin.get(db, :nonexistent, default: :not_found)
# => :not_found
Goblin.get(db, :alice, tag: :admins)
# => "Alice"
@spec get_and_update( :gen_statem.server_ref(), term(), (term() -> {term(), term()}), keyword() ) :: term()
Updates a key and returns a value.
Updates a value corresponding to key via the provided function and can return an arbitrary value.
The provided function gets the previous value (nil if previously not set) as the argument.
The provided function must return a two-tuple with the first element as the return value and the second element as the updated value.
Parameters
db- The database server (PID or registered name)key- The key to updateupdater- A function that updates the valueopts- A keyword list with the following options (default:[])::tag- Tag to namespace the keys under:timeout- Timeout (in milliseconds) for the calls (default::infinity)
Returns
- Reply from the provided function
Examples
Goblin.get_and_update(db, :alice, fn name ->
{String.upcase(name), name <> "alice"}
end)
# => "ALICE"
@spec get_and_update_multi( :gen_statem.server_ref(), [term()], (map() -> {term(), map()}), keyword() ) :: term()
Updates multiple keys and returns a value.
Gets and updates multiple keys in a single transaction via a provided function.
Any keys that are not already present in the database receive the value set in the :default option, if provided, otherwise excluded.
The provided function must return a two-tuple with the first element as the return value and the second element as the updated value.
Parameters
db- The database server (PID or registered name)keys- The key to updateupdater- A function that updates the valueopts- A keyword list with the following options (default:[])::tag- Tag to namespace the keys under:default- Default value for non-existing keys:timeout- Timeout (in milliseconds) for the calls (default::infinity)
Returns
- Reply from the provided function
Examples
Goblin.get_and_update_multi(db, [:alice, :bob, :charlie], fn entries ->
entries = Enum.into(entries, %{}, fn {key, val} -> {key, String.upcase(val)} end)
{Map.keys(entries), entries}
end)
# => [:alice, :bob, :charlie]
Retrieves values for multiple keys in a single read.
Keys not found in the database are excluded from the result.
Raises Goblin.IOError if the underlying storage cannot be read.
Parameters
db- The database server (PID or registered name)keys- A list of keys to look upopts- A keyword list with the following options (default:[])::tag- Tag the keys are namespaced under
Returns
- A list of
{key, value}tuples for keys found, in unspecified order
Examples
Goblin.get_multi(db, [:alice, :bob])
# => [{:alice, "Alice"}, {:bob, "Bob"}]
Goblin.get_multi(db, [:alice, :nonexistent])
# => [{:alice, "Alice"}]
Checks whether a key is a member of the database or not.
False positives
If the key has been flushed to disk, then membership is checked via the disk table's Bloom filters, i.e. it can yield a false positive in some cases.
Parameters
db- The database server (PID or registered name)key- The key to check membership offopts- A keyword list with the following options (default:[])::tag- Tag the keys are namespaced under
Returns
- A boolean indicating if the key exists or not
Examples
Goblin.has_key?(db, :alice)
# => false
Goblin.put(db, :alice, "Alice")
Goblin.has_key?(db, :alice)
# => true
Writes a key-value pair to the database.
Parameters
db- The database server (PID or registered name)key- Any Elixir term to use as the keyvalue- Any Elixir term to be associated withkeyopts- A keyword list with the following options (default:[])::tag- Tag to namespace the key under:timeout- Timeout (in milliseconds) for the calls (default::infinity)
Returns
:ok
Examples
Goblin.put(db, :alice, "Alice")
# => :ok
Goblin.put(db, :alice, "Alice", tag: :admins)
# => :ok
@spec put_multi(:gen_statem.server_ref(), Enumerable.t({term(), term()}), keyword()) :: :ok
Writes multiple key-value pairs in a single transaction.
Parameters
db- The database server (PID or registered name)pairs- A list of{key, value}tuplesopts- A keyword list with the following options (default:[])::tag- Tag to namespace the keys under:timeout- Timeout (in milliseconds) for the calls (default::infinity)
Returns
:ok
Examples
Goblin.put_multi(db, [{:alice, "Alice"}, {:bob, "Bob"}, {:charlie, "Charlie"}])
# => :ok
Performs a read-only transaction.
A snapshot is taken to provide a consistent mvcc of the database. Multiple readers run concurrently without blocking each other. Attempting to write within a read transaction raises.
Raises Goblin.IOError if the underlying storage cannot be read.
Parameters
db- The database server (PID or registered name)callback- A function that takes aGoblin.Tx.t()struct
Returns
- The return value of
callback
Examples
Goblin.read(db, fn tx ->
alice = Goblin.Tx.get(tx, :alice)
bob = Goblin.Tx.get(tx, :bob)
{alice, bob}
end)
# => {"Alice", "Bob"}
Removes a key from the database.
Parameters
db- The database server (PID or registered name)key- The key to removeopts- A keyword list with the following options (default:[])::tag- Tag the key is namespaced under:timeout- Timeout (in milliseconds) for the calls (default::infinity)
Returns
:ok
Examples
Goblin.remove(db, :alice)
# => :ok
Goblin.get(db, :alice)
# => nil
Removes multiple keys from the database in a single transaction.
Parameters
db- The database server (PID or registered name)keys- A list of keys to removeopts- A keyword list with the following options (default:[])::tag- Tag the keys are namespaced under:timeout- Timeout (in milliseconds) for the calls (default::infinity)
Returns
:ok
Examples
Goblin.remove_multi(db, [:alice, :bob, :charlie])
# => :ok
Returns a stream of key-value pairs, optionally bounded by a range. Captures snapshots at enumeration.
Entries are sorted by key in ascending order.
Both min and max are inclusive.
Raises Goblin.IOError if the underlying storage cannot be read.
Parameters
db- The database server (PID or registered name)opts- Keyword list of options::min- Minimum key, inclusive (optional):max- Maximum key, inclusive (optional):tag- Tag to filter by (optional)
Returns
- A stream of
{key, value}tuples
Examples
Goblin.scan(db) |> Enum.to_list()
# => [{:alice, "Alice"}, {:bob, "Bob"}, {:charlie, "Charlie"}]
Goblin.scan(db, min: :bob) |> Enum.to_list()
# => [{:bob, "Bob"}, {:charlie, "Charlie"}]
Goblin.scan(db, min: :alice, max: :bob) |> Enum.to_list()
# => [{:alice, "Alice"}, {:bob, "Bob"}]
scan = Goblin.scan(db)
Enum.to_list(scan)
# => []
Goblin.put(db, :alice, "Alice")
Enum.to_list(scan)
# => [{:alice, "Alice"}]
Starts the database, see start_link/1 for more details.
Starts the database.
Creates the data_dir if it does not exist.
Options
:name- Registered name for the database (optional, defaults toGoblin):data_dir- Directory path for database files (required):mem_limit- Bytes to buffer in memory before flushing to disk (default: 64 MB):bf_fpp- Bloom filter false positive probability (default: 0.01)
Returns
{:ok, pid}- On successful start{:error, reason}- On failure
Examples
{:ok, db} = Goblin.start_link(
name: MyApp.DB,
data_dir: "/var/lib/myapp/db"
)
@spec stop(:gen_statem.server_ref(), term(), timeout()) :: :ok
Stops the database.
@spec transaction( :gen_statem.server_ref(), (Goblin.Tx.t() -> {:commit, Goblin.Tx.t(), term()} | {:error, :aborted}), keyword() ) :: term()
Executes a read-write transaction.
Transactions are executed serially and are ACID-compliant.
The provided function receives a transaction struct and must return
{:commit, tx, reply} to commit, or {:abort, reply} to abort.
Calling transaction or any of its derivatives (put, put_multi, remove, remove_multi) on the same database from within a transaction raises.
Parameters
db- The database server (PID or registered name)callback- A function that takes aGoblin.Tx.t()and returns a transaction resultopts- A keyword list with options::timeout- Timeout (in milliseconds) for the calls (default::infinity)
Returns
reply- The reply from{:commit, tx, reply}when committed{:error, :aborted}- When the transaction is aborted
Examples
Goblin.transaction(db, fn tx ->
counter = Goblin.Tx.get(tx, :counter, default: 0)
tx
|> Goblin.Tx.put(:counter, counter + 1)
|> Goblin.Tx.commit()
end)
# => :ok
Goblin.transaction(db, fn tx ->
tx
|> Goblin.Tx.abort()
end)
# => :error
@spec update( :gen_statem.server_ref(), term(), (term() -> term()) | (term(), term() -> term()), keyword() ) :: :ok
Updates a key.
Updates the value corresponding to key either via the provided function or via default.
See update_multi for more information.
Parameters
db- The database server (PID or registered name)key- The key to updateupdater- A function that updates the valueopts- A keyword list with the following options (default:[])::tag- Tag to namespace the keys under:default- Default value if key does not already exist:timeout- Timeout (in milliseconds) for the calls (default::infinity)
Returns
:ok
Examples
Goblin.update(db, :counter, &(&1 + 1), default: 1)
# => :ok
@spec update_multi( :gen_statem.server_ref(), [term()], (term() -> term()) | (term(), term() -> term()), keyword() ) :: {:ok, non_neg_integer()}
Updates multiple keys.
Updates multiple values corresponding to multiple keys via the provided function.
The update function can be of either arity 1 or 2.
With arity 1, the function receives only the previous value.
With arity 2, the function receives both the key and the previous value.
For any non-existing keys, they are inserted with the :default option.
If :default is not provided, then non-existing keys are not inserted.
Parameters
db- The database server (PID or registered name)keys- The keys to updateupdater- A function that updates the valueopts- A keyword list with the following options (default:[])::tag- Tag to namespace the keys under:default- Default value for non-existing keys:timeout- Timeout (in milliseconds) for the calls (default::infinity)
Returns
{:ok, no_entries_updated}
Examples
Goblin.update_multi(db, [:alice, :bob, :charlie], &String.upcase(&1))
# => {:ok, 3}
Goblin.update_multi(db, [:alice, :bob, :charlie], fn k, v ->
if k == :bob,
do: String.upcase(v),
else: String.downcase(v)
end)
# => {:ok, 3}