ZeroIPC - Active Computational Substrate for Shared Memory¶

What is ZeroIPC?¶

ZeroIPC transforms shared memory from passive storage into an active computational substrate, enabling both imperative and functional programming paradigms across process boundaries. It provides zero-copy data sharing with sophisticated concurrency primitives, reactive streams, and codata structures—bringing modern programming abstractions to inter-process communication.

Key Features¶

• Zero-Copy Performance - Direct memory access without serialization overhead
• Language Independence - True parallel implementations in C, C++, and Python (not bindings!)
• Lock-Free Concurrency - Atomic operations and CAS-based algorithms for high-performance synchronization
• Minimal Metadata - Only store name/offset/size for maximum flexibility
• Duck Typing - Runtime type specification (Python) or compile-time templates (C++)
• Simple Discovery - Named structures for easy cross-process lookup
• Reactive Programming - Functional reactive streams with map, filter, fold operators
• Codata Support - Futures, lazy evaluation, and infinite streams
• CSP Concurrency - Channels for synchronous message passing
• Comprehensive CLI Tools - Virtual filesystem interface for inspection and debugging

Quick Example¶

C++ Producer¶

#include <zeroipc/memory.h>
#include <zeroipc/array.h>

// Create shared memory segment
zeroipc::Memory mem("/sensor_data", 10*1024*1024);  // 10MB

// Create typed array
zeroipc::Array<float> temps(mem, "temperature", 1000);
temps[0] = 23.5f;
temps[1] = 24.1f;

Python Consumer¶

from zeroipc import Memory, Array
import numpy as np

# Open same shared memory segment
mem = Memory("/sensor_data")

# Read with duck typing - user specifies type
temps = Array(mem, "temperature", dtype=np.float32)
print(temps[0])  # 23.5
print(temps[1])  # 24.1

Why ZeroIPC?¶

Traditional IPC mechanisms force you to choose between performance and ease of use:

• Sockets/Pipes: Easy to use but slow (serialization overhead)
• Message Queues: Safe but limited (fixed message sizes)
• Raw Shared Memory: Fast but difficult (manual synchronization, no type safety)

ZeroIPC gives you all three: performance, safety, and ease of use.

Performance Comparison¶

Zero-copy means truly zero overhead

Data Structures¶

Traditional Structures¶

• Array - Fixed-size contiguous storage with atomic operations
• Queue - Lock-free MPMC circular buffer using CAS
• Stack - Lock-free LIFO with ABA-safe operations
• Map - Lock-free hash map with linear probing
• Set - Lock-free hash set for unique elements
• Pool - Object pool with free list management
• Ring - High-performance ring buffer for streaming

Synchronization Primitives¶

• Semaphore - Cross-process counting/binary semaphore
• Barrier - Multi-process synchronization barrier
• Latch - One-shot countdown synchronization

Codata & Computational Structures¶

• Future - Asynchronous computation results across processes
• Lazy - Deferred computations with automatic memoization
• Stream - Reactive data flows with FRP operators
• Channel - CSP-style synchronous/buffered message passing

Use Cases¶

ZeroIPC excels at:

• High-frequency sensor data sharing
• Multi-process simulations and scientific computing
• Real-time analytics pipelines
• Cross-language data processing
• Zero-copy producer-consumer patterns
• Reactive event processing

Architecture Highlights¶

Binary Format Specification¶

All language implementations follow the same binary format:

[Table Header][Table Entries][Data Structure 1][Data Structure 2]...

• Table Header: Magic number, version, entry count, next offset
• Table Entry: Name (32 bytes), offset (4 bytes), size (4 bytes)
• Data Structures: Raw binary data, layout determined by structure type

Language Equality¶

Unlike traditional IPC libraries where one language is "primary":

• Both languages can create - Python and C++ can both allocate new structures
• Both languages can read - Either can discover and access existing structures
• Type safety per language - C++ uses templates, Python uses NumPy dtypes
• No bindings needed - Each implementation stands alone

Getting Started¶

Ready to dive in? Here's your roadmap:

1. Installation - Get ZeroIPC up and running
2. Quick Start - Your first shared memory program in 5 minutes
3. Tutorial - Step-by-step guide to all features
4. CLI Tool - Learn the virtual filesystem interface

Project Status¶

ZeroIPC is actively developed and production-ready:

• Current Version: 2.0
• Stability: Stable API, comprehensive test suite
• Performance: 200x test suite optimization (20 min → 2 min)
• Coverage: All core structures implemented across C, C++, and Python
• Testing: Fast/Medium/Slow/Stress test categorization with CTest labels

Community¶

• GitHub: Report issues and contribute
• Documentation: You're reading it!
• Examples: See the examples directory

License¶

MIT License - See LICENSE file for details