Getting Started

This guide will help you get up and running with libmagic-rs, whether you want to use it as a CLI tool or integrate it into your Rust applications.

Installation

Prerequisites

• Rust 1.89+ (2024 edition)
• Git for cloning the repository
• Cargo (comes with Rust)

From Source

Currently, libmagic-rs is only available from source as it’s in early development:

# Clone the repository
git clone https://github.com/EvilBit-Labs/libmagic-rs.git
cd libmagic-rs

# Build the project
cargo build --release

# Run tests to verify installation
cargo test

The compiled binary will be available at target/release/rmagic.

Development Build

For development or testing the latest features:

# Clone and build in debug mode
git clone https://github.com/EvilBit-Labs/libmagic-rs.git
cd libmagic-rs
cargo build

# The debug binary is at target/debug/rmagic

Quick Start

CLI Usage

# Identify files using built-in rules (no external magic file needed)
./target/release/rmagic --use-builtin example.bin

# JSON output format
./target/release/rmagic --use-builtin --json example.bin

# Use a custom magic file
./target/release/rmagic --magic-file /usr/share/misc/magic example.bin

# Multiple files
./target/release/rmagic --use-builtin file1.bin file2.pdf file3.zip

# Read from stdin
echo -ne '\x7fELF' | ./target/release/rmagic --use-builtin -

# Help and options
./target/release/rmagic --help

Library Usage

Add libmagic-rs to your Cargo.toml:

[dependencies]
libmagic-rs = { git = "https://github.com/EvilBit-Labs/libmagic-rs.git" }

Basic usage with built-in rules (no external files needed):

use libmagic_rs::{LibmagicError, MagicDatabase};

fn main() -> Result<(), LibmagicError> {
    // Use built-in rules compiled into the binary
    let db = MagicDatabase::with_builtin_rules()?;

    // Evaluate a file
    let result = db.evaluate_file("example.bin")?;
    println!("File type: {}", result.description);
    println!("Confidence: {}", result.confidence);

    // Evaluate an in-memory buffer
    let buffer = b"\x7fELF\x02\x01\x01\x00";
    let result = db.evaluate_buffer(buffer)?;
    println!("Buffer type: {}", result.description);

    Ok(())
}

Project Structure

Understanding the project layout will help you navigate the codebase:

libmagic-rs/
├── Cargo.toml              # Project configuration
├── src/
│   ├── lib.rs              # Library API (MagicDatabase, EvaluationConfig, etc.)
│   ├── main.rs             # CLI implementation (rmagic binary)
│   ├── error.rs            # Error types (LibmagicError, ParseError, EvaluationError)
│   ├── parser/
│   │   ├── mod.rs          # Magic file parser entry point
│   │   ├── ast.rs          # AST data structures
│   │   ├── grammar.rs      # nom-based parsing combinators
│   │   ├── loader.rs       # File/directory loading with format detection
│   │   └── format.rs       # Magic file format detection
│   ├── evaluator/
│   │   ├── mod.rs          # Evaluation engine
│   │   ├── offset.rs       # Offset resolution
│   │   ├── operators.rs    # Comparison operators with cross-type coercion
│   │   └── types.rs        # Type interpretation with endianness
│   ├── output/
│   │   ├── mod.rs          # Output types and conversion
│   │   └── json.rs         # JSON/JSON Lines formatting
│   ├── io/
│   │   └── mod.rs          # Memory-mapped I/O (FileBuffer)
│   ├── mime.rs             # MIME type mapping
│   ├── tags.rs             # Semantic tag extraction
│   └── builtin_rules.rs    # Pre-compiled magic rules
├── tests/                  # Integration tests
├── third_party/            # Canonical libmagic tests and magic files
└── docs/                   # This documentation

Development Setup

If you want to contribute or modify the library:

1. Clone and Setup

git clone https://github.com/EvilBit-Labs/libmagic-rs.git
cd libmagic-rs

# Install development dependencies
cargo install cargo-nextest  # Faster test runner
cargo install cargo-watch    # Auto-rebuild on changes

2. Development Workflow

# Check code without building
cargo check

# Run tests (fast)
cargo nextest run

# Run tests with coverage
cargo test

# Format code
cargo fmt

# Lint code (strict mode)
cargo clippy -- -D warnings

# Build documentation
cargo doc --open

3. Continuous Development

# Auto-rebuild and test on file changes
cargo watch -x check -x test

# Auto-run specific tests
cargo watch -x "test ast_structures"

Current Capabilities

• AST Data Structures: Complete implementation with full serialization
• Magic File Parser: nom-based parser for magic file DSL with hierarchical rules
• Rule Evaluator: Engine for executing rules against files with graceful error handling
• Memory-Mapped I/O: Efficient file access with comprehensive bounds checking
• CLI Tool (rmagic): Full-featured CLI with text/JSON output, stdin, timeouts, and built-in rules
• Built-in Rules: Pre-compiled detection for common file types (ELF, ZIP, PDF, JPEG, PNG, etc.)
• MIME Type Mapping: Opt-in MIME type detection
• Output Formatters: Text and JSON output with tag enrichment
• Strength Calculation: Rule priority scoring with !:strength directives
• Confidence Scoring: Match confidence based on rule hierarchy depth
• Timeout Protection: Configurable per-file evaluation timeouts
• Build System: Cargo configuration with strict clippy pedantic linting
• Testing: 940+ comprehensive tests across all modules
• Documentation: This guide, API documentation, and architecture docs

Example Magic Rules

You can parse magic rules from text or work with AST structures directly:

Parsing Magic Files

#![allow(unused)]
fn main() {
use libmagic_rs::parser::parse_text_magic_file;

// Parse a simple magic file
let magic_content = r#"
ELF file format
0 string \x7fELF ELF executable
>4 byte 1 32-bit
>4 byte 2 64-bit
"#;

let rules = parse_text_magic_file(magic_content)?;
assert_eq!(rules.len(), 1);
assert_eq!(rules[0].children.len(), 2);
}

Working with AST Directly

#![allow(unused)]
fn main() {
use libmagic_rs::parser::ast::*;

// Create a simple ELF detection rule
let elf_rule = MagicRule {
    offset: OffsetSpec::Absolute(0),
    typ: TypeKind::Byte,
    op: Operator::Equal,
    value: Value::Uint(0x7f), // First byte of ELF magic
    message: "ELF executable".to_string(),
    children: vec![],
    level: 0,
};

// Serialize to JSON for inspection
let json = serde_json::to_string_pretty(&elf_rule)?;
println!("{}", json);
}

Evaluating Rules

#![allow(unused)]
fn main() {
use libmagic_rs::evaluator::{evaluate_rules_with_config, EvaluationContext};
use libmagic_rs::parser::ast::*;
use libmagic_rs::EvaluationConfig;

// Create a rule to detect ELF files
let rule = MagicRule {
    offset: OffsetSpec::Absolute(0),
    typ: TypeKind::Byte,
    op: Operator::Equal,
    value: Value::Uint(0x7f),
    message: "ELF magic".to_string(),
    children: vec![],
    level: 0,
};

// Evaluate against a buffer
let buffer = &[0x7f, 0x45, 0x4c, 0x46]; // ELF magic bytes
let config = EvaluationConfig::default();
let matches = evaluate_rules_with_config(&[rule], buffer, &config)?;

assert_eq!(matches.len(), 1);
assert_eq!(matches[0].message, "ELF magic");
}

Testing Your Setup

Verify everything is working correctly:

# Run all tests
cargo test

# Run specific AST tests
cargo test ast_structures

# Check code quality
cargo clippy -- -D warnings

# Verify documentation builds
cargo doc

# Test CLI
cargo run -- README.md

Next Steps

1. Explore the AST: Check out AST Data Structures to understand the core types
2. Read the Architecture: See Architecture Overview for the big picture
3. Follow Development: Watch the GitHub repository for updates
4. Contribute: See Development Setup for contribution guidelines

Getting Help

• Documentation: This guide covers all current functionality
• API Reference: Run cargo doc --open for detailed API docs
• Issues: Report bugs or request features
• Discussions: Ask questions or share ideas

The project is in active development, so check back regularly for new features and capabilities!