Goblin (Goblin v0.13.0)

Copy Markdown View Source

A lightweight, embedded, LSM-tree database for Elixir.

Goblin is a persistent key-value store with ACID transactions, crash recovery, and automatic background compaction. It runs inside your application's supervision tree.

Starting a database

{:ok, db} = Goblin.start_link(
  name: MyApp.DB,
  data_dir: "/path/to/db"
)

Basic operations

Goblin.put(db, :alice, "Alice")
Goblin.get(db, :alice)
# => "Alice"

Goblin.remove(db, :alice)
Goblin.get(db, :alice)
# => nil

Batch operations

Goblin.put_multi(db, [{:alice, "Alice"}, {:bob, "Bob"}])

Goblin.get_multi(db, [:alice, :bob])
# => [{:alice, "Alice"}, {:bob, "Bob"}]

Transactions

Goblin.transaction(db, fn tx ->
  counter = Goblin.Tx.get(tx, :counter, default: 0)

  tx
  |> Goblin.Tx.put(:counter, counter + 1)
  |> Goblin.Tx.commit()
end)
# => :ok

See start_link/1 for configuration options.

Summary

Functions

Compare and swap a key.

Returns whether background compaction is currently running.

Exports a snapshot of the database as a .tar.gz archive.

Returns whether a memory-to-disk flush is currently running.

Retrieves the value associated with a key.

Updates a key and returns a value.

Updates multiple keys and returns a value.

Retrieves values for multiple keys in a single read.

Checks whether a key is a member of the database or not.

Writes a key-value pair to the database.

Writes multiple key-value pairs in a single transaction.

Performs a read-only transaction.

Removes a key from the database.

Removes multiple keys from the database in a single transaction.

Returns a stream of key-value pairs, optionally bounded by a range. Captures snapshots at enumeration.

Starts the database, see start_link/1 for more details.

Starts the database.

Executes a read-write transaction.

Functions

cas(db, key, old, new, opts \\ [])

@spec cas(:gen_statem.server_ref(), term(), term(), term(), keyword()) :: boolean()

Compare and swap a key.

Compare and swap a value corresponding to key. Returns true if swapped, false otherwise.

Comparison is done via ==.

Parameters

  • db - The database server (PID or registered name)
  • key - The key to update
  • old - The value to compare with
  • new - The value to swap to
  • opts - A keyword list with the following options (default: []):
    • :tag - Tag to namespace the keys under
    • :timeout - Timeout (in milliseconds) for the calls (default: :infinity)

Returns

  • true if swapped
  • false if not swapped

Examples

Goblin.cas(db, :alice, "alice", "ALICE")
# => true

child_spec(opts)

@spec child_spec(keyword()) :: Supervisor.child_spec()

compacting?(db, opts \\ [])

@spec compacting?(
  :gen_statem.server_ref(),
  keyword()
) :: boolean()

Returns whether background compaction is currently running.

export(db, export_dir, opts \\ [])

@spec export(:gen_statem.server_ref(), Path.t(), keyword()) ::
  {:ok, Path.t()} | {:error, term()}

Exports a snapshot of the database as a .tar.gz archive.

The archive can be unpacked and used as the data_dir for a new database instance, acting as a backup.

The export is run inside the server, thus blocking file deletion and writes until completed.

Parameters

  • db - The database server (PID or registered name)
  • export_dir - Directory to place the exported .tar.gz file

Returns

  • {:ok, export_path} - Path to the created archive
  • {:error, reason} - If an error occurred

Examples

Goblin.export(db, "/backups")
# => {:ok, "/backups/goblin_20260220T120000Z.tar.gz"}

flushing?(db, opts \\ [])

@spec flushing?(
  :gen_statem.server_ref(),
  keyword()
) :: boolean()

Returns whether a memory-to-disk flush is currently running.

get(db, key, opts \\ [])

Retrieves the value associated with a key.

Returns the default value if the key is not found.

Raises Goblin.IOError if the underlying storage cannot be read.

Parameters

  • db - The database server (PID or registered name)
  • key - The key to look up
  • opts - A keyword list with the following options (default: []):
    • :tag - Tag the key is namespaced under
    • :default - Value to return if key is not found (default: nil)

Returns

  • The value associated with the key, or default if not found

Examples

Goblin.get(db, :alice)
# => "Alice"

Goblin.get(db, :nonexistent)
# => nil

Goblin.get(db, :nonexistent, default: :not_found)
# => :not_found

Goblin.get(db, :alice, tag: :admins)
# => "Alice"

get_and_update(db, key, updater, opts \\ [])

@spec get_and_update(
  :gen_statem.server_ref(),
  term(),
  (term() -> {term(), term()}),
  keyword()
) ::
  term()

Updates a key and returns a value.

Updates a value corresponding to key via the provided function and can return an arbitrary value. The provided function gets the previous value (nil if previously not set) as the argument. The provided function must return a two-tuple with the first element as the return value and the second element as the updated value.

Parameters

  • db - The database server (PID or registered name)
  • key - The key to update
  • updater - A function that updates the value
  • opts - A keyword list with the following options (default: []):
    • :tag - Tag to namespace the keys under
    • :timeout - Timeout (in milliseconds) for the calls (default: :infinity)

Returns

  • Reply from the provided function

Examples

Goblin.get_and_update(db, :alice, fn name -> 
  {String.upcase(name), name <> "alice"} 
end)
# => "ALICE"

get_and_update_multi(db, keys, updater, opts \\ [])

@spec get_and_update_multi(
  :gen_statem.server_ref(),
  [term()],
  (map() -> {term(), map()}),
  keyword()
) ::
  term()

Updates multiple keys and returns a value.

Gets and updates multiple keys in a single transaction via a provided function. Any keys that are not already present in the database receive the value set in the :default option, if provided, otherwise excluded. The provided function must return a two-tuple with the first element as the return value and the second element as the updated value.

Parameters

  • db - The database server (PID or registered name)
  • keys - The key to update
  • updater - A function that updates the value
  • opts - A keyword list with the following options (default: []):
    • :tag - Tag to namespace the keys under
    • :default - Default value for non-existing keys
    • :timeout - Timeout (in milliseconds) for the calls (default: :infinity)

Returns

  • Reply from the provided function

Examples

Goblin.get_and_update_multi(db, [:alice, :bob, :charlie], fn entries -> 
  entries = Enum.into(entries, %{}, fn {key, val} -> {key, String.upcase(val)} end)
  {Map.keys(entries), entries}
end)
# => [:alice, :bob, :charlie]

get_multi(db, keys, opts \\ [])

Retrieves values for multiple keys in a single read.

Keys not found in the database are excluded from the result.

Raises Goblin.IOError if the underlying storage cannot be read.

Parameters

  • db - The database server (PID or registered name)
  • keys - A list of keys to look up
  • opts - A keyword list with the following options (default: []):
    • :tag - Tag the keys are namespaced under

Returns

  • A list of {key, value} tuples for keys found, in unspecified order

Examples

Goblin.get_multi(db, [:alice, :bob])
# => [{:alice, "Alice"}, {:bob, "Bob"}]

Goblin.get_multi(db, [:alice, :nonexistent])
# => [{:alice, "Alice"}]

has_key?(db, key, opts \\ [])

Checks whether a key is a member of the database or not.

False positives

If the key has been flushed to disk, then membership is checked via the disk table's Bloom filters, i.e. it can yield a false positive in some cases.

Parameters

  • db - The database server (PID or registered name)
  • key - The key to check membership off
  • opts - A keyword list with the following options (default: []):
    • :tag - Tag the keys are namespaced under

Returns

  • A boolean indicating if the key exists or not

Examples

Goblin.has_key?(db, :alice)
# => false
Goblin.put(db, :alice, "Alice")
Goblin.has_key?(db, :alice)
# => true

put(db, key, val, opts \\ [])

Writes a key-value pair to the database.

Parameters

  • db - The database server (PID or registered name)
  • key - Any Elixir term to use as the key
  • value - Any Elixir term to be associated with key
  • opts - A keyword list with the following options (default: []):
    • :tag - Tag to namespace the key under
    • :timeout - Timeout (in milliseconds) for the calls (default: :infinity)

Returns

  • :ok

Examples

Goblin.put(db, :alice, "Alice")
# => :ok

Goblin.put(db, :alice, "Alice", tag: :admins)
# => :ok

put_multi(db, pairs, opts \\ [])

@spec put_multi(:gen_statem.server_ref(), Enumerable.t({term(), term()}), keyword()) ::
  :ok

Writes multiple key-value pairs in a single transaction.

Parameters

  • db - The database server (PID or registered name)
  • pairs - A list of {key, value} tuples
  • opts - A keyword list with the following options (default: []):
    • :tag - Tag to namespace the keys under
    • :timeout - Timeout (in milliseconds) for the calls (default: :infinity)

Returns

  • :ok

Examples

Goblin.put_multi(db, [{:alice, "Alice"}, {:bob, "Bob"}, {:charlie, "Charlie"}])
# => :ok

read(db, callback)

Performs a read-only transaction.

A snapshot is taken to provide a consistent mvcc of the database. Multiple readers run concurrently without blocking each other. Attempting to write within a read transaction raises.

Raises Goblin.IOError if the underlying storage cannot be read.

Parameters

  • db - The database server (PID or registered name)
  • callback - A function that takes a Goblin.Tx.t() struct

Returns

  • The return value of callback

Examples

Goblin.read(db, fn tx ->
  alice = Goblin.Tx.get(tx, :alice)
  bob = Goblin.Tx.get(tx, :bob)
  {alice, bob}
end)
# => {"Alice", "Bob"}

remove(db, key, opts \\ [])

Removes a key from the database.

Parameters

  • db - The database server (PID or registered name)
  • key - The key to remove
  • opts - A keyword list with the following options (default: []):
    • :tag - Tag the key is namespaced under
    • :timeout - Timeout (in milliseconds) for the calls (default: :infinity)

Returns

  • :ok

Examples

Goblin.remove(db, :alice)
# => :ok

Goblin.get(db, :alice)
# => nil

remove_multi(db, keys, opts \\ [])

Removes multiple keys from the database in a single transaction.

Parameters

  • db - The database server (PID or registered name)
  • keys - A list of keys to remove
  • opts - A keyword list with the following options (default: []):
    • :tag - Tag the keys are namespaced under
    • :timeout - Timeout (in milliseconds) for the calls (default: :infinity)

Returns

  • :ok

Examples

Goblin.remove_multi(db, [:alice, :bob, :charlie])
# => :ok

scan(db, opts \\ [])

Returns a stream of key-value pairs, optionally bounded by a range. Captures snapshots at enumeration.

Entries are sorted by key in ascending order. Both min and max are inclusive.

Raises Goblin.IOError if the underlying storage cannot be read.

Parameters

  • db - The database server (PID or registered name)
  • opts - Keyword list of options:
    • :min - Minimum key, inclusive (optional)
    • :max - Maximum key, inclusive (optional)
    • :tag - Tag to filter by (optional)

Returns

  • A stream of {key, value} tuples

Examples

Goblin.scan(db) |> Enum.to_list()
# => [{:alice, "Alice"}, {:bob, "Bob"}, {:charlie, "Charlie"}]

Goblin.scan(db, min: :bob) |> Enum.to_list()
# => [{:bob, "Bob"}, {:charlie, "Charlie"}]

Goblin.scan(db, min: :alice, max: :bob) |> Enum.to_list()
# => [{:alice, "Alice"}, {:bob, "Bob"}]

scan = Goblin.scan(db)
Enum.to_list(scan)
# => []
Goblin.put(db, :alice, "Alice")
Enum.to_list(scan)
# => [{:alice, "Alice"}]

start(opts)

Starts the database, see start_link/1 for more details.

start_link(opts)

Starts the database.

Creates the data_dir if it does not exist.

Options

  • :name - Registered name for the database (optional, defaults to Goblin)
  • :data_dir - Directory path for database files (required)
  • :mem_limit - Bytes to buffer in memory before flushing to disk (default: 64 MB)
  • :bf_fpp - Bloom filter false positive probability (default: 0.01)

Returns

  • {:ok, pid} - On successful start
  • {:error, reason} - On failure

Examples

{:ok, db} = Goblin.start_link(
  name: MyApp.DB,
  data_dir: "/var/lib/myapp/db"
)

stop(db, reason \\ :normal, timeout \\ :infinity)

@spec stop(:gen_statem.server_ref(), term(), timeout()) :: :ok

Stops the database.

transaction(db, callback, opts \\ [])

@spec transaction(
  :gen_statem.server_ref(),
  (Goblin.Tx.t() -> {:commit, Goblin.Tx.t(), term()} | {:error, :aborted}),
  keyword()
) :: term()

Executes a read-write transaction.

Transactions are executed serially and are ACID-compliant. The provided function receives a transaction struct and must return {:commit, tx, reply} to commit, or {:abort, reply} to abort.

Calling transaction or any of its derivatives (put, put_multi, remove, remove_multi) on the same database from within a transaction raises.

Parameters

  • db - The database server (PID or registered name)
  • callback - A function that takes a Goblin.Tx.t() and returns a transaction result
  • opts - A keyword list with options:
    • :timeout - Timeout (in milliseconds) for the calls (default: :infinity)

Returns

  • reply - The reply from {:commit, tx, reply} when committed
  • {:error, :aborted} - When the transaction is aborted

Examples

Goblin.transaction(db, fn tx ->
  counter = Goblin.Tx.get(tx, :counter, default: 0)
  tx
  |> Goblin.Tx.put(:counter, counter + 1)
  |> Goblin.Tx.commit()
end)
# => :ok

Goblin.transaction(db, fn tx ->
  tx
  |> Goblin.Tx.abort()
end)
# => :error

update(db, key, updater, opts \\ [])

@spec update(
  :gen_statem.server_ref(),
  term(),
  (term() -> term()) | (term(), term() -> term()),
  keyword()
) :: :ok

Updates a key.

Updates the value corresponding to key either via the provided function or via default. See update_multi for more information.

Parameters

  • db - The database server (PID or registered name)
  • key - The key to update
  • updater - A function that updates the value
  • opts - A keyword list with the following options (default: []):
    • :tag - Tag to namespace the keys under
    • :default - Default value if key does not already exist
    • :timeout - Timeout (in milliseconds) for the calls (default: :infinity)

Returns

  • :ok

Examples

Goblin.update(db, :counter, &(&1 + 1), default: 1)
# => :ok

update_multi(db, keys, updater, opts \\ [])

@spec update_multi(
  :gen_statem.server_ref(),
  [term()],
  (term() -> term()) | (term(), term() -> term()),
  keyword()
) :: {:ok, non_neg_integer()}

Updates multiple keys.

Updates multiple values corresponding to multiple keys via the provided function. The update function can be of either arity 1 or 2. With arity 1, the function receives only the previous value. With arity 2, the function receives both the key and the previous value. For any non-existing keys, they are inserted with the :default option. If :default is not provided, then non-existing keys are not inserted.

Parameters

  • db - The database server (PID or registered name)
  • keys - The keys to update
  • updater - A function that updates the value
  • opts - A keyword list with the following options (default: []):
    • :tag - Tag to namespace the keys under
    • :default - Default value for non-existing keys
    • :timeout - Timeout (in milliseconds) for the calls (default: :infinity)

Returns

  • {:ok, no_entries_updated}

Examples

Goblin.update_multi(db, [:alice, :bob, :charlie], &String.upcase(&1))
# => {:ok, 3}

Goblin.update_multi(db, [:alice, :bob, :charlie], fn k, v ->
  if k == :bob,
    do: String.upcase(v),
    else: String.downcase(v)
end)
# => {:ok, 3}