Open-source Python library

Experiment analysis in Python.

A/B testing and causal inference, from a dataframe or your warehouse. Turn experiment data into estimates, intervals, and decisions.

See the result

Experiment: Pricing Page v2
Illustrative values · fixed-horizon intervals
Metric Lift
Revenue per User +3.4%
[+0.9%, +5.9%]
Conversion +1.2%
[-0.3%, +2.7%]
D14 Retention +2.1%
[+0.4%, +3.8%]
-5.0%0.0%+5.0%
Experiment: Pricing Page v2
Illustrative values · fixed-horizon intervals
Metric Lift
Revenue per User +3.4%
[+0.9%, +5.9%]
Conversion +1.2%
[-0.3%, +2.7%]
D14 Retention +2.1%
[+0.4%, +3.8%]
-5.0%0.0%+5.0%
Synthetic example · illustrative values, not a real experiment.
Start with a dataframe GitHub
$ pip install 'increment @ git+https://github.com/kylejcaron/increment.git'

Development documentation · Python 3.12–3.14. The current API is available from source, not the PyPI release. See installation and optional extras.

01 / Dataframes

From a dataframe. No backend.

One row per unit is all it takes: no YAML, no warehouse connection, not even DuckDB.

Run your first analysis
analysis.py One row per unit
from increment import Analysis

results = Analysis.from_unit_summary(
    df,
    unit="user_id",
    group="variant",
    control="control",
    metrics={"revenue": "mean", "converted": "conversion"},
).run()

02 / Warehouses

Same engine. Your warehouse.

Declare experiments, metrics, and fact sources in YAML. Increment builds the queries and runs them on DuckDB, Snowflake, BigQuery, or Postgres via Ibis.

Explore warehouse analysis
warehouse.py Definitions → results
from increment import Analysis

# con is an Ibis connection to your warehouse
results = Analysis.from_definitions(
    experiment_name="checkout_redesign",
    definitions_path="definitions/",
    con=con,
).run()

03 / Segments

The average isn't the whole story.

Break results out by customer segment. Compare lifts and intervals to see where the treatment effect differs.

Explore segment effects
Pricing Page v2 · by segment
Illustrative values · 3 segments · 14-day daily trend to final read
Metric Segment Lift % Lift Daily lift
Revenue per User New users +1.0%
[-2.1%, +4.1%]
Returning +3.6%
[+1.0%, +6.2%]
Enterprise +5.8%
[+2.6%, +9.0%]
-5.0%0.0%+5.0% 36912Jan
Pricing Page v2 · by segment
Illustrative values · 3 segments · 14-day daily trend to final read
Metric Segment Lift % Lift Daily lift
Revenue per User New users +1.0%
[-2.1%, +4.1%]
Returning +3.6%
[+1.0%, +6.2%]
Enterprise +5.8%
[+2.6%, +9.0%]
-5.0%0.0%+5.0% 36912Jan
Illustrative output with synthetic values, not a real experiment.

04 / Personalization

Find who benefits most.

Go beyond average lift. Estimate how treatment effects vary with customer characteristics, score new customer profiles, and build targeting rules from experiment data.

Explore personalization
Illustrative estimates and intervals, not experiment results. Engagement is measured before the experiment. Relative lift divides each profile's conversion change by its fixed 10% control conversion rate.
Conditional treatment effects
Relative lift in purchase conversion (%)
Profile Relative lift
New customer Low engagement
New customer High engagement
Returning customer Low engagement
Returning customer High engagement
0%+10%+20%
Conditional treatment effects
Relative lift in purchase conversion (%)
Profile Relative lift
New customer Low engagement
New customer High engagement
Returning customer Low engagement
Returning customer High engagement
0%+10%+20%

05 / Statistical methods

Methods with their assumptions stated.

CUPED variance reduction, sample-ratio-mismatch checks, multiple-comparison corrections, power planning, sequential monitoring under stated assumptions, and IPTW, DML, or AIPW for observational treatments. Not every method works with every metric or design, and unsupported combinations are refused rather than silently changed. See the compatibility guide and statistical limitations.

from increment import Analysis, Method, MetricSpec

results = Analysis.from_unit_summary(
    df,  # one row per unit, with a pre-experiment revenue column
    unit="user_id",
    group="variant",
    control="control",
    metrics=[MetricSpec(name="revenue", type="mean", covariate="pre_revenue")],
).run(
    decision_method=Method(name="unadjusted"),
    sensitivity_methods=(Method(name="cuped", variance_reduction="cuped"),),
)