Reproducibility and Random Seeds#

This tutorial explains how to make probabilistic generation reproducible in probLM-solver.

Why this matters#

Sampling-based decoding is stochastic. If you do not control randomness, outputs may vary between runs even with the same prompt, model, and hyperparameters.

The package provides a RandomManager to coordinate deterministic random streams from one root seed.

Minimal reproducible script#

 1from pathlib import Path
 2
 3from problm_solver.samplers import SampleLowTemp, adjust_identity
 4from problm_solver.llama_interface import Model
 5from problm_solver.random import RandomManager
 6
 7model_path = Path.home() / '.problm-solver' / 'models' / 'Qwen3.5-0.8B-Q4_K_M.gguf'
 8prompt = 'Why is the sky blue?'
 9
10# One seed for the whole run.
11run_rng = RandomManager(seed=12345)
12
13model = Model(
14    fname=str(model_path),
15    context=prompt,
16    logits_all=True,
17    rng=run_rng,
18)
19
20out_identity = model.generate_with_sampler(
21    top_k=30,
22    top_p=0.9,
23    adjust_fn=adjust_identity,
24    max_tokens=128,
25    sampling_method='identity',
26)
27
28model.change_context(prompt)
29
30out_low_temp = model.generate_with_sampler(
31    top_k=30,
32    top_p=0.9,
33    adjust_fn=SampleLowTemp(alpha=2.0),
34    max_tokens=128,
35    sampling_method='low_temp_alpha2',
36)
37
38print('identity:', ''.join(out_identity.response_probabilities[0]))
39print('low-temp:', ''.join(out_low_temp.response_probabilities[0]))

Running this script again with the same seed reproduces the same outputs (assuming same model file, package version, and runtime environment).

RNG inputs accepted by user-facing APIs#

Methods that consume randomness accept rng as an RNGLike value:

  • numpy.random.Generator: use your own generator directly.

  • int: interpreted as a seed.

  • RandomManager: use named deterministic streams from one root seed.

  • None: use default non-deterministic behavior.

Example with explicit per-call seed:

text = model.query(rng=2026)

Example with user-managed generator:

import numpy as np

gen = np.random.default_rng(2026)
text_a = model.query(rng=gen)
text_b = model.query(rng=gen)

Notes#

For strict reproducibility across machines, pin:

  • model file/version,

  • problm_solver version,

  • Python/numpy versions,

  • and any hardware-specific inference settings.