Dataset Version Control for Quant Traders: A Practical Guide

Quant researcher comparing market data revisions

Dataset version control means recording exactly which data a result was built from, so you can rebuild that result months later and get the same numbers. For a trader, that means every backtest points to a fixed, identifiable snapshot of your price history, not to a folder that quietly changed. Below: the tools, a layout that works, what to record, and the traps specific to market data.

TL;DR

Version the data, the code, the parameters and the output together. Versioning only the files is half the job.
Keep large files out of plain Git. Use a pointer-based tool such as DVC, or at minimum immutable, hashed, read-only snapshots.
Treat every vendor correction, rollover rule or cleaning step as a new dataset version with a written changelog.
Point-in-time thinking matters: a dataset that is silently revised or survivor-filtered will flatter your backtest.
Never overwrite a dataset. Add a new version and keep the old one for audits.

What dataset version control actually solves
Why plain Git is the wrong place for market data
The four things to version together
A folder and naming layout that holds up
What to put in the changelog
Market-data traps that versioning exposes
Step by step: a minimal setup in an afternoon
Connecting versions to live results
Our perspective
Starting from a clean, ready-to-import base
Common mistakes to avoid
FAQ
Sources

What dataset version control actually solves

Software teams solved this for code decades ago. Datasets are harder because they are large, binary and often replaced in place. You download a fresh minute-bar file, drop it over the old one, rerun the strategy and the equity curve is different. Was it the code? The parameters? The data? Without versions you cannot say.

Three questions should always be answerable:

Which exact data produced this result?
What changed between this dataset and the previous one?
Can I rerun the experiment and get the same output?

The DVC user guide frames the goal as reproducibility: it describes an open-source tool for data management, pipeline automation and experiment management that helps teams manage large datasets and make projects reproducible. The same guide describes working with three principles: human-readable metafiles, versioning through Git, and cloud-based data collaboration.

Why plain Git is the wrong place for market data

Git is excellent for text. A multi-year, multi-instrument set of minute bars is neither small nor text-friendly. Pushing raw data into a repository bloats it and slows everything down, and hosting platforms enforce repository size limits. The DVC documentation describes the alternative: the Git repository holds small pointer files with hashes and metadata, while the heavy files live in external storage such as S3 or GCS.

That pointer idea is the core of most dataset version control tools:

Git versions the pointer, the code and the config.
Remote storage holds the actual bytes, addressed by content hash.
A checkout command restores the data that matches any commit.

If you have never used it, the DVC tutorial on data and model versioning walks through managing multiple datasets and models hands-on.

A short history that explains the design

Older centralized systems relied on locking files and a constant server connection. Kitware's overview of distributed version control describes how distributed systems unseated them with concurrent repositories and network-free operation. For your data this matters in one practical way: you want every research machine to hold a complete, verifiable history, not depend on one shared drive that someone can overwrite.

What version control is not

Project trackers are not dataset versioning. Jira, for example, links issues to release versions) such as 1.0 or 2.1.1, but it does not store or version the data or code files themselves. Use it for tasks if you like. It will not tell you which CSV fed last Tuesday's test.

The four things to version together

A dataset snapshot alone does not reproduce a backtest. The DVC guide's holistic view is that you version the raw inputs, but also the configuration parameters, processing scripts and resulting models. For a quant, that translates to four layers:

Layer	Examples	How to version it
Raw data	Minute bars per symbol, rollover tables	Hashed, read-only snapshots, tracked by pointer files
Processing	Cleaning scripts, resampling, timezone conversion	Git, tagged with the data version
Parameters	Spreads, commissions, slippage, strategy inputs	Config files in Git
Results	Trade lists, equity curves, platform reports	Stored per run, named by commit and data version

The same logic gives you an audit trail. If a result looks too good, you can walk back through each layer and find which one moved.

A folder and naming layout that holds up

You do not need heavy infrastructure on day one. This structure works with or without a dedicated tool:

project/
  data/
    raw/          # never edited, one folder per version
      v2026.10/
      v2026.07/
    processed/    # generated only by scripts
  config/         # spreads, costs, session times
  src/            # cleaning and strategy code
  results/        # one subfolder per run
  CHANGELOG.md    # what changed in each data version

Rules that keep it honest:

Raw is read-only. If a bar looks wrong, you fix it in a script that writes to processed, and the script is versioned.
Version by date or counter, never "final" or "new". Names like final_v2_REAL are the problem version control exists to prevent.
Record a checksum. A SHA-256 hash per file, stored in a manifest, proves the file is unchanged.
Stamp every result. Each run writes the code commit, the data version and the config hash into its report.

What to put in the changelog

A data changelog is short and specific. For each new version, note:

Date of download and source.
Instruments and date range covered.
Time zone and session convention.
Any bars removed, repaired or interpolated, and why.
Contract roll or back-adjustment method for futures-style series.
Differences from the previous version, such as a corrected spike or a filled gap.

If you do your own cleaning, document the rule as well as the result. Our walkthrough of an intraday data cleaning spike test shows the kind of explicit, repeatable rule that belongs in this log, and the 7 step data audit for MT4 and MT5 is a sensible checklist to run before you tag a version as trusted.

Market-data traps that versioning exposes

Silent revisions and look-ahead bias

Data can change after the fact. Fundamental figures get restated, and prices get corrected. If your backtest uses the revised value for a date when only the original existed, it knows the future. Campbell Harvey's backtesting protocol paper notes that truly out-of-sample data does not exist when history is finite and known, which is a reason to be strict about what a model could have seen at each point.

The Seven Sins of Quantitative Investing describes the ideal remedy as a point-in-time database, where the vendor archives the originally reported data with reporting dates and tracks later revisions. Your own version history does the same job at the file level: version 2026.07 is what you knew in July, and it stays that way.

Survivorship bias

If a dataset only contains instruments that still exist, failures vanish and results inflate. A study of one widely used fundamentals database reported that about 8.5% of firms disappeared each year between 1980 and 2018, so omitting them skews results. The Your Backtest Is Cheating article argues that a historical screen is an as-of query over a versioned data system, not a filter over rows, and warns about reused tickers. That framing is the right mental model for any dataset you keep.

For forex, metals and index minute bars, survivorship is less of an issue than gaps, timestamps, rollover handling and spikes. The principle is identical: know exactly what is in the file, and keep that knowledge attached to a version.

Survivorship bias — overview diagram

Quiet changes in the same filename

The most common real-world failure is mundane. A file is refreshed, a time zone is shifted, a gap is filled and nobody records it. Our piece on intraday data sources practitioners must audit covers what to check when a source changes, and the MQL4 data prep and per-year gap audit shows how a year-by-year view reveals holes that an overall quality figure hides.

Step by step: a minimal setup in an afternoon

Create a Git repository for code, config and the changelog.
Initialize a data tracking tool such as DVC, and add your raw data folder. The tool writes small pointer files you commit to Git.
Configure remote storage, whether a cloud bucket or a network drive, and push the data.
Tag the commit with the data version, for example data-2026.10.
Define your pipeline (clean, resample, backtest) as scripted stages so outputs regenerate from inputs.
Write the manifest: file names, sizes, hashes, date range.
Run a baseline backtest and store its report under the tag.
When data changes, add a new version, update the changelog, rerun the baseline and compare it to the previous report before trusting any new result.

Step 8 is the one people skip. Rerunning a fixed baseline on each new data version tells you immediately whether the data moved your numbers.

Connecting versions to live results

Versioned data is also what lets you compare a backtest with what happened in production. If the two diverge, you need to rule out the data first, then costs, then execution. A backtest versus live trading comparison on 200 verified trades shows the kind of signal-to-fill matching this requires, and it only works if the backtest side can be reproduced exactly. Costs belong in versioned config too, which is why we suggest testing three transaction cost regimes as separate, named configurations.

Our perspective

We collect and maintain minute-bar data, and we see the same pattern in support conversations: a trader's strategy "stopped working" after a data refresh, and nobody can say what changed. The cause is rarely exotic. It is usually an undocumented difference between two files.

That is why we verify our datasets bar by bar, document adjustments and keep datasets maintained for direct MT4 and MT5 import. A clean dataset is the foundation, but version control is what keeps it trustworthy over time. Our view:

Never replace a dataset in place. Add a version.
A cleaning rule you cannot write down is a rule you cannot reproduce.
Data quality standards should hold for human researchers and for automated, AI-driven backtesting alike, because a model will happily overfit to a defect.
Rerun a baseline after every data change. It takes minutes and prevents weeks of confusion.

— Start

Starting from a clean, ready-to-import base

Versioning only helps if the first version is sound. If you would rather not clean raw feeds yourself, our historical data sets cover forex, metals, stock indices, bonds and commodities as minute bars since 2014, ready to import into MT4 and MT5. You can browse forex data or stock indices directly. Our Annual Plan is listed at 119 EUR per year. Download a set, store it as your first immutable version, record its hash and begin your changelog from there.

Common mistakes to avoid

Versioning only the strategy code. The data decides the outcome as much as the logic.
Storing results without their inputs. An equity curve with no data version is an anecdote.
Editing raw files by hand. Even a one-bar fix needs a script and a log entry.
Mixing time zones across versions. Record the convention each time.
Keeping only the latest version. You lose the ability to explain old results.
Skipping portfolio-level checks. Multi-instrument tests multiply the places data can drift, as shown in our guide to MT4 portfolio backtesting with export replay.

FAQ

What is the difference between Git and dataset version control tools?

Git tracks text changes line by line and struggles with large binary files. Dataset tools such as DVC keep small pointer files with hashes in Git and store the heavy data in external storage. You get the same commit and checkout workflow without bloating the repository.

Is Jira a version control tool?

No. Jira is a project and issue tracker that can group issues into release versions, but it does not store or version data or code files. Use Git, with a pointer-based tool for large data, for that job.

How did people do version control before Git?

They used centralized systems such as CVS and Subversion, which followed earlier lock-based tools like RCS. These typically needed a connection to a central server, which is a main reason distributed systems replaced them for most development work.

Do I need a tool, or are dated folders enough?

For a solo trader with a few datasets, read-only dated folders, a hash manifest and a changelog are a solid start. Move to a tool when you share data across machines or people, or when you want pipelines that rebuild results automatically.

Sources

Related resources

Explore BacktestMarket's Expert Advisor robots to put the ideas in this article into practice.