PrimeBackup-v1.13.0.pyz

News

File Chunking

Docs: https://tisunion.github.io/PrimeBackup/chunking/

Added file chunking support, designed for large files that are frequently partially modified

When enabled, Prime Backup splits a file into chunks. Each chunk is hashed, compressed, and stored independently. Unchanged chunks are reused across backups, greatly improving deduplication for partially modified files

Chunked files still have a normal blob record for the whole file, but the chunk payloads are stored as entries in pack files under <storage_root>/packs/

Supported algorithms:

• fastcdc_32k, fastcdc_128k, fastcdc_1m: Content-Defined Chunking via FastCDC, with average chunk sizes of 32 KiB, 128 KiB, and 1 MiB
• fixed_4k, fixed_32k, fixed_128k, fixed_1m: fixed-size chunking with fixed byte-offset boundaries
• fixed_auto: adaptive fixed-size chunking that uses the previous same-path backup to switch between 128 KiB and 4 KiB granularity (alpha)

File chunking is still a beta feature and is currently intended for early access use only. Don't worry, data created with chunking will remain fully compatible and accessible in future versions

New backup config options:

• backup.chunking_enabled (bool, default false): whether to enable chunking
• backup.chunking_rules (list): ordered list of chunking rules. Each rule has:
  • algorithm: one of the chunking algorithms listed above
  • file_size_threshold (int): minimum file size in bytes to trigger this rule
  • patterns (list): gitignore-style path patterns

The default value contains two rules:

• fixed_128k fixed-size chunking for .log files larger than 10 MiB
• fixed_auto chunking for .mca files larger than 256 KiB

FastCDC algorithms require the optional dependency pyfastcdc, installable via pip install -r requirements.optional.txt or by installing pyfastcdc directly. Fixed-size algorithms have no extra dependency

Pack File Storage and Compaction

Added pack files for chunk storage. Instead of storing each chunk as a separate file, new chunk payloads are written as entries inside pack files

Pack files are append-only. When chunks are deleted, the byte ranges they occupied become dead space. Prime Backup can reclaim that space by compacting pack files: live entries are rewritten into new pack files, then the old pack files are removed

New backup config options:

• backup.pack_auto_compact_threshold (float, default 0.5): if a pack's live-size ratio drops below this threshold after chunk deletion, it is compacted immediately
• backup.pack_maintenance_compact_threshold (float, default 0.8): threshold used by maintenance tasks such as scheduled pack compaction and database prune

New database config:

• database.compact_pack: scheduled pack compaction job, enabled by default, with default crontab 0 5 * * 0

New database command:

• !!pb database compact_packs: compact all pack files with a 1.0 threshold, skipping only fully live pack files

New Backup Config: mutating_file_patterns

• Added backup.mutating_file_patterns config option (gitignore-style list, default [])
• Files matching these patterns are treated as frequently mutating files, such as mod files that may keep changing after /save-off
• Prime Backup skips pre-hash computation for these files and always copies them to a temporary location before processing, ensuring a stable snapshot and speeding up backup creation for this kind of file

New Database Commands

• Added !!pb database reassign_backup_id [<order>]: reassign all backup IDs sequentially based on the given sort order (id, id_r, time, time_r)
  • This operation is irreversible and requires confirmation
• Added !!pb database inspect chunk <id_or_hash>: inspect detailed information of a chunk
• Added !!pb database inspect chunk_group <id_or_hash>: inspect detailed information of a chunk group
• Added !!pb database inspect pack <id_or_file_name>: inspect detailed information of a pack file by pack ID or unique pack file-name prefix
• Added !!pb database validate chunks: validate chunks, chunk groups, bindings, pack references, and chunk payload integrity
• Added !!pb database validate packs: validate pack file existence, recorded size, and live statistics

New CLI Commands

• Added make: create a backup from the command line
• Added back: restore a backup from the command line
• Added init: initialize a new Prime Backup database at the directory given by --db

The CLI now supports -c/--config for loading config.json. If not provided, it tries to auto-detect ../config/prime_backup/config.json relative to the database directory. Commands that need user backup settings, such as make and back, require a config file

Documentation

• Added dedicated chunking documentation pages for overview, CDC chunking, and fixed-size chunking
• Added maintenance documentation for corrupted blob handling
• Added maintenance documentation explaining backup prune, database prune, SQLite vacuum, pack compaction, and base fileset shrink

Changes

• Default hash method changed from xxh128 to blake3
  • blake3 is now a required dependency; the blake3 lib published its first stable release in 2024 with up-to-date prebuilt wheels, so installation should not be blocked by Rust tooling
  • This affects new installations only; existing databases keep their recorded hash method unless a hash migration is performed
• pyfastcdc was added to requirements.optional.txt as an optional dependency for CDC chunking
• Database schema v4 now stores hash columns as binary data internally while keeping Python-side values as hex strings
• Chunked blob storage now uses pack entries, and backup creation, import, export, restore, FUSE, validation, hash migration, and compression migration have been updated for pack-backed chunks
• Backup timestamp precision improved from microseconds to nanoseconds again
• Backup file extension matching during import is now case-insensitive, for example .TAR.GZ is recognized
• !!pb database overview output is now structured into [Backup], [File], [Blob], [Chunk], and [Pack] sections
• !!pb database overview now includes blob store totals, overall compression ratio, file/blob deduplication stats, chunk stats, and pack live-size stats
• !!pb database inspect blob now accepts an integer ID or a hash prefix, and shows storage-method-specific information
• !!pb database prune now also deletes unknown pack files and runs pack compaction using backup.pack_maintenance_compact_threshold
• !!pb database validate all now includes both packs and chunks
• !!pb database validate now accepts multiple parts in one command, for example !!pb database validate blobs chunks packs, and shows a summary after multi-part validation
• Scheduled database backup now supports database.backup.max_amount (default 100) and deletes older database backup archives after a new database backup is created
• Backup creation logs now include the total source file size and periodic progress for slow file creation
• Backup diff output now shows uid and gid on unix system, old/new affected data sizes in the summary, and file-level hash differences for changed files
• Added error log and a "need to restart the server manually" warning when a backup restore fails

Fixes

• Fixed config.prune.timezone_override having no effect
• Fixed an unhandled error during backup import when metadata was invalid, which previously crashed instead of suggesting --auto-meta
• Fixed --no-meta having no effect on certain FUSE operations
• Fixed possible database migration failure on Windows when the DB file was still held open by Prime Backup
• A lot more fixes and improvements were made, see the full commit history for details