Skip to content

Extending and persisting compiled tries

This document explains how compiled Radixor tries can be reopened, extended with domain vocabulary, rebuilt, and stored for deployment. The Java API is described here; the Python (PyO3) runtime offers the equivalent through radixor.TrieBuilder (see Customizing a Dictionary). The resulting compiled Radixor model can be loaded by Java, Python (PyO3), and Python-C.

Reopen and extend a compiled trie

FrequencyTrieBuilders.copyOf(...) reconstructs a mutable builder from a compiled trie. The reconstructed builder preserves the key-local value counts of the compiled trie as currently stored, making it suitable for subsequent modification and recompilation. Reconstruction is performed from the compiled state, not from the original unreduced insertion history.

import java.io.IOException;
import java.nio.file.Path;

import org.egothor.stemmer.FrequencyTrie;
import org.egothor.stemmer.FrequencyTrieBuilders;
import org.egothor.stemmer.CompiledPatchCommand;
import org.egothor.stemmer.PatchCommandEncoder;
import org.egothor.stemmer.ReductionMode;
import org.egothor.stemmer.ReductionSettings;
import org.egothor.stemmer.StemmerPatchTrieBinaryIO;

public final class ExtendCompiledStemmerExample {

    private ExtendCompiledStemmerExample() {
        throw new AssertionError("No instances.");
    }

    public static void main(final String[] arguments) throws IOException {
        final FrequencyTrie<String> compiledTrie = StemmerPatchTrieBinaryIO.read(
                Path.of("stemmers", "english.radixor.gz"));

        final ReductionSettings settings = ReductionSettings.withDefaults(
                ReductionMode.MERGE_SUBTREES_WITH_EQUIVALENT_RANKED_GET_ALL_RESULTS);

        final FrequencyTrie.Builder<String> builder = FrequencyTrieBuilders.copyOf(
                compiledTrie,
                String[]::new,
                settings);

        final String word = "microservices";
        final String stem = "microservice";
        final PatchCommandEncoder encoder = PatchCommandEncoder.builder()
                .traversalDirection(compiledTrie.traversalDirection())
                .build();
        final String patch = encoder.encode(word, stem);
        builder.putDominant(word, patch);

        final FrequencyTrie<String> updatedTrie = builder.build();

        final String storedPatch = updatedTrie.get(word);
        final String actualStem = CompiledPatchCommand
                .compile(storedPatch, updatedTrie.traversalDirection())
                .apply(word);
        System.out.println(word + " -> " + actualStem);

        StemmerPatchTrieBinaryIO.write(
                updatedTrie,
                Path.of("stemmers", "english-custom.radixor.gz"));
    }
}

This enables a layered workflow:

  1. start from a bundled or already compiled stemmer,
  2. reconstruct a builder,
  3. encode each new word-to-stem relationship as a patch command and add it,
  4. compile and persist a new binary artifact.

The insertion key is the observed word (microservices), while the stored value is the encoded transformation to its desired stem (microservice). Do not copy a patch string from another word: generate it with PatchCommandEncoder so offsets and traversal direction remain correct. Add the canonical stem as a no-op relationship too when the custom vocabulary must recognize it as an input in its own right.

The compiled representation is a directed acyclic graph (DAG): semantically equivalent subtrees may share one physical node, whose frequencies have already been aggregated. Reconstruction retains that provenance. Consequently, an unmodified rebuild counts a shared node once instead of multiplying its counts by the number of incoming logical paths. Modifying one expanded path creates a copy-on-write reduction boundary, so its new values and frequencies cannot leak into unchanged paths that previously shared the node. The original unreduced per-path insertion history is not recoverable; the compiled aggregate is the authoritative starting state.

Choosing how an added rule wins

A compiled model generalizes suffixes: the English model contracts the -s plural into one rule that strips a trailing s from any word, so a rule added for one specific word can be shadowed by that generalization. Two things control whether your rule takes effect — the update method you use on the builder, and the lookup mode you read with. copyOf reconstructs the contracted generalizations faithfully, so an unmodified round-trip reproduces the original stemmer and your additions do not weaken the model's coverage of other words.

Update methods

put(key, value, count) accumulates a frequency. Because the returned stem is the highest-frequency value at a node, a word that already has a rule can out-rank a single added occurrence. The additional update methods make the intent explicit; all return the builder for chaining.

Method Effect at the key's node
put(key, value, count) accumulate a raw frequency
putDominant(key, value) make value the dominant result, keeping other values as lower-ranked alternatives
set(key, value) replace every value at the node with value
putIfAbsent(key, value) store value only when the node has no value yet
remove(key) delete every value at the exact node
remove(key, value) delete one value, keeping the rest

putDominant is the usual choice for overriding one rule while keeping the prior candidate visible in getAll; set discards the alternatives entirely.

Concrete before-and-after example

The following generic trie uses readable values so the local frequency and replacement semantics are visible. A stemmer builder stores encoded patch commands instead, but the six update methods behave identically.

import org.egothor.stemmer.FrequencyTrie;
import org.egothor.stemmer.ReductionMode;

final FrequencyTrie.Builder<String> builder =
        new FrequencyTrie.Builder<>(String[]::new,
                ReductionMode.MERGE_SUBTREES_WITH_EQUIVALENT_RANKED_GET_ALL_RESULTS);

builder.put("token", "legacy", 3)
        .put("token", "alternate");
final FrequencyTrie<String> initial = builder.build();
// initial.getEntries("token") ==
//     [ValueCount[value=legacy, count=3], ValueCount[value=alternate, count=1]]

builder.put("token", "alternate");
final FrequencyTrie<String> accumulated = builder.build();
// "legacy" still wins 3 to 2: put accumulates; it does not promise dominance.

builder.putDominant("token", "alternate");
final FrequencyTrie<String> promoted = builder.build();
// "alternate" now wins 4 to 3, while "legacy" remains in getAll("token").

builder.remove("token", "legacy");
final FrequencyTrie<String> oneCandidate = builder.build();
// Only "alternate" remains, with its count of 4.

builder.set("token", "curated")
        .putIfAbsent("token", "ignored");
final FrequencyTrie<String> replaced = builder.build();
// set discarded every old value and stored "curated" with count 1;
// putIfAbsent then did nothing because that node was not empty.

builder.remove("missing")                 // missing key: no-op
        .remove("token", "not-present")  // missing value: no-op
        .remove("token")                  // exact node becomes empty
        .putIfAbsent("token", "fallback");
final FrequencyTrie<String> refilled = builder.build();
// refilled.get("token") returns "fallback".

// Every earlier build is an immutable snapshot. Later builder changes did not
// alter initial, accumulated, promoted, oneCandidate, or replaced.

The resulting states are:

Operation Values at exact node token, in lookup order
put("legacy", 3) then put("alternate") legacy:3, alternate:1
another put("alternate") legacy:3, alternate:2
putDominant("alternate") alternate:4, legacy:3
remove("legacy") alternate:4
set("curated") then putIfAbsent("ignored") curated:1
remove(key) then putIfAbsent("fallback") fallback:1

The table abbreviates the Java calls by omitting the repeated "token" key. Counts belong to one exact trie node. Removing a value does not decrement it; it removes that candidate completely. set always resets the replacement count to one. putDominant sets its value to one more than the highest other count, so calling it again can change the count only when another candidate has become equally or more frequent.

Composition and edge cases

  • Calls are immediately applied to the mutable builder and therefore compose left to right. There is no implicit transaction or rollback.
  • build() creates a read-only snapshot and leaves the builder usable. A later edit never mutates an earlier snapshot.
  • null keys and values are rejected. put(..., count) requires a positive count and fails on integer overflow.
  • The empty string is a valid key and addresses the root node; it is not treated as a missing key.
  • Keys are normalized according to the builder metadata. Updates and lookups must therefore use compatible case and diacritic policies.
  • remove(key) removes values only from the exact node. It does not prune the path and does not remove a shorter rule that happens to match the same input.
  • remove(key, value) preserves every other candidate and its count. Missing keys or values are deliberate no-ops, which makes idempotent cleanup safe.

Reading with the specific rule

An added rule sits at the word's own deep node, beneath the shallow generalization. Under the default LookupMode.FIRST the generalization short-circuits and the added rule is not seen; read with LookupMode.LAST so the specific rule wins (see Querying and Ambiguity Handling).

import org.egothor.stemmer.LookupMode;

final String word = "windows";
final String patch = PatchCommandEncoder.builder()
        .traversalDirection(compiledTrie.traversalDirection())
        .build()
        .encode(word, word); // identity
builder.putDominant(word, patch);

final FrequencyTrie<String> updated = builder.build();
final String stored = updated.withLookupMode(LookupMode.LAST).get(word);
// 'stored' is the identity rule, so the word is not rewritten by the -s generalization.

remove(key) targets the word's exact node. A word that stems only through a shorter contracted generalization has no value of its own to delete, so removing the full word is a no-op — override it with set or putDominant and read with LookupMode.LAST instead, or remove the shorter suffix key, which affects every word it covers.

Persist and deploy compiled tries

StemmerPatchTrieBinaryIO reads and writes patch-command tries as GZip-compressed binary files. StemmerPatchTrieLoader exposes convenience methods around the same persistence functionality.

import java.io.IOException;
import java.nio.file.Path;

import org.egothor.stemmer.StemmerPatchTrieBinaryIO;

StemmerPatchTrieBinaryIO.write(trie, Path.of("stemmers", "english.radixor.gz"));

In deployment terms, the cleanest model is usually:

  • compile once,
  • persist the binary artifact,
  • load the artifact directly in runtime services.

Binary-first operational model

For larger dictionaries or controlled deployment environments, a binary-first workflow is usually the most robust choice:

  • prepare the compiled trie offline,
  • keep the preparation step outside the runtime startup path,
  • version and distribute the binary artifact,
  • load the finished trie directly in production.

This model works especially well when domain-specific extensions are added in layers and then recompiled into a new read-only artifact.

Continue with

Inspecting persisted metadata

After loading a compiled artifact, applications can inspect the persisted build descriptor directly:

final FrequencyTrie<CompiledPatchCommand> trie =
        StemmerPatchTrieLoader.loadBinaryCompiled("build/stemmers/cs_cz.dat.gz");
final TrieMetadata metadata = trie.metadata();

System.out.println(metadata.formatVersion());
System.out.println(metadata.traversalDirection());
System.out.println(metadata.reductionSettings().reductionMode());
System.out.println(metadata.diacriticProcessingMode());

This is especially useful when a deployment manages multiple artifacts compiled under different traversal or reduction regimes.