About TamaDB

TamaDB is a basic document-oriented NoSQL database engine sitting at the core of data exchange within a smart-endpoints/dumb-pipes architecture. The term "basic" reflects its intentionally limited ability to perform complex operations at the document level — it is designed primarily as a high-performance cache for transferring JSON documents between data producers and consumers. The database stores JSON documents in a tree-like structure with unique identifiers, supports atomic attribute-level updates with history tracking, and provides multiple retrieval patterns. The system offers three document models (Closed, Open, and Hybrid) with optional JSON schema enforcement for data integrity.

What Problem Does TamaDB Solve?

In modern data architectures, systems need to exchange structured data between producers (sensors, applications, services) and consumers (dashboards, analytics, downstream services) with minimal latency and maximum reliability.

TamaDB is a high-performance transportation cache — a purpose-built intermediary that:

• Receives JSON documents from multiple producers via dedicated channels
• Validates documents against registered JSON schemas
• Stores temporarily with configurable time-to-live (TTL)
• Pushes updates to registered consumers in real-time via webhooks

TamaDB is not a general-purpose database. It is intentionally limited in scope: no complex queries, no joins, no aggregations. Its strength is speed, simplicity, and reliable data delivery.

Architectural Position

TamaDB and PamaDB together form the Transmission Layer — the intermediary between producers and consumers:

How the Transmission Layer works:

• Producers send data into the Transmission Layer via TamaDB (the entry point)
• TamaDB validates, versions, stores, and can deliver directly to consumers
• PamaDB extends TamaDB for delivery requirements outside TamaDB's scope (transformations, frozen documents, publishing)
• Consumers receive data from either TamaDB or PamaDB, depending on the delivery criteria
• Delivery criteria can be defined by either the producer or the consumer

Key principle: Dumb-pipe role. TamaDB never alters document content. It validates, stores, versions, and delivers — nothing more. PamaDB handles any transformation needs.

Core Concepts

Document

A JSON object stored in TamaDB. Documents are structured as trees of Bunches (objects) and Leaves (terminal values). Each leaf is independently versioned and has its own history.

Document_Type

The central organising concept. Every document belongs to exactly one Document_Type, which defines: the Master_Schema (what fields are valid and required), the document model (how strictly the schema is enforced), Channels (ingestion endpoints), TTL (document lifespan), and Subscriptions (notification targets).

Channels

An ingestion endpoint tied to a Document_Type. Producers push data to a channel URL. Each channel can have its own Channel_Schema (a subset of the Master_Schema), enrichment bindings (TamaDB injects fields the producer doesn't send), and status control (active, inactive, or blocked).

Upsert & Merge Semantics

TamaDB has a single ingestion operation — upsert. There is no separate "create" vs "update". If the document doesn't exist, it's created. If it exists, incoming fields are merged into the stored document. Fields present in the payload update (previous value goes to history). Fields absent from the payload are preserved unchanged. No data is ever removed by a store operation.

Subscriptions

Consumers register webhooks (callback URLs) and receive real-time updates. Subscriptions can be at the Document_Type level (fires for any document of that type) or D-Id level (fires for a specific document only). Payload is snapshotted at the moment of update — delivery is decoupled from the store.

Document TTL

Every document has a time-to-live. TamaDB is a cache, not an archive. TTL hierarchy: Document > Document_Type > Instance (most specific wins). Expired documents are soft-deleted. TTL=0 means "process and deliver, but don't cache" (transient).

Schema Validation

TamaDB validates documents on ingestion using JSON Schema (Draft 7). Required fields must be present, field types must match, and unknown fields are rejected under the Document Closed Model (DCM). Validation happens on the Enriched_Document (payload + channel-supplied fields).

Document Models

TamaDB vs Alternatives

Increment 1 — What's Delivered

Capabilities

Technology Stack

Roadmap

Increment 2

• HDOM (Hybrid Document Open Model) — schema validation for known fields, extensions allowed
• Discrete leaf updates — update individual fields by path without sending full document
• Foreign D-Ids (full) — dynamic/rule-based identity derivation
• Library Sections — category-based document grouping
• Subscription updates — in-place modification of active subscriptions
• Schema adoption — propagate mode auto-adopts latest Retrieval_Schema version

Increment 3

• DOM (Document Open Model) — no schema enforcement, fully dynamic
• Alien D-Ids — documents with no identity (immutable, non-updatable one-shots)
• Element-level arrays — address individual array elements (sparse transmission)

Future Vision

• PamaDB — data publishing layer (transformations, frozen documents, hard subscriptions)
• Sanitation — validate stored documents against template schemas
• Multi-protocol — gRPC, WebSocket ingestion alongside REST
• Idempotent updates — detect and skip duplicate payloads
• Storage segmentation — per-Document_Type data isolation

End-to-End Data Flow

TamaDB User Manual

Getting Started

Prerequisites

• Node.js 22 or later
• npm (comes with Node.js)

Installation & Startup

# Clone the repository
git clone https://tamagitlab.duckdns.org:8443/teneris/tamadb.git
cd tamadb

# Install dependencies
npm ci

# Build
npm run build

# Start the server
npm start
# Or for development (auto-reload):
npm run dev

TamaDB starts on port 3000 by default. Verify with:

curl http://localhost:3000/health

Response:

{
  "status": "ok",
  "version": "0.1.0",
  "storageMode": "auto",
  "logLevel": "debug"
}

Configuration (Environment Variables)

Step 1: Register a Schema

Before storing documents, define the structure they must follow.

curl -X POST http://localhost:3000/api/v1/schemas \
  -H "Content-Type: application/json" \
  -d '{
    "ref": "temperature-sensor-v1",
    "schema": {
      "$schema": "http://json-schema.org/draft-07/schema#",
      "type": "object",
      "properties": {
        "sensorId": { "type": "string", "identityKind": "primary-key" },
        "location": { "type": "string" },
        "temperature": { "type": "number" },
        "timestamp": { "type": "string" }
      },
      "required": ["sensorId", "location", "temperature"]
    }
  }'

Response:

{ "ref": "temperature-sensor-v1", "version": 1 }

Key points:

• identityKind: "primary-key" tells TamaDB which field(s) form the document's identity
• required fields must be present in every ingested document (or supplied by enrichment)

Step 2: Register a Document_Type

Group documents under a named type with its Master_Schema.

curl -X POST http://localhost:3000/api/v1/config/document-types \
  -H "Content-Type: application/json" \
  -d '{
    "name": "temperature-readings",
    "masterSchema": "temperature-sensor-v1",
    "model": "closed",
    "ttl": 3600
  }'

Response:

{
  "name": "temperature-readings",
  "masterSchema": "temperature-sensor-v1",
  "model": "closed",
  "ttl": 3600,
  "channels": [],
  "createdAt": "2026-07-17T10:00:00.000Z",
  "updatedAt": "2026-07-17T10:00:00.000Z"
}

Parameters:

• model: "closed" — strict validation, no unknown fields allowed
• ttl: 3600 — documents expire after 1 hour

Step 3: Register a Channel (Optional)

Channels give producers their own ingestion endpoint, optionally with enrichment.

curl -X POST http://localhost:3000/api/v1/config/channels \
  -H "Content-Type: application/json" \
  -d '{
    "channelId": "garage-sensor",
    "documentType": "temperature-readings"
  }'

Add Enrichment Bindings

Enrichment injects fields the producer doesn't need to send:

curl -X PUT http://localhost:3000/api/v1/config/channels/garage-sensor/enrichment \
  -H "Content-Type: application/json" \
  -d '{
    "bindings": [
      { "fieldPath": "location", "staticValue": "Garage" },
      { "fieldPath": "sensorId", "staticValue": "GARAGE-TEMP-01" }
    ]
  }'

Now the producer only needs to send {"temperature": 22.5} — TamaDB fills in the rest.

Step 4: Ingest Documents

Direct ingestion (Document_Type-level)

curl -X POST http://localhost:3000/api/v1/ingest/temperature-readings \
  -H "Content-Type: application/json" \
  -d '{
    "document": {
      "sensorId": "LIVING-ROOM-01",
      "location": "Living Room",
      "temperature": 21.5,
      "timestamp": "2026-07-17T10:05:00Z"
    }
  }'

Response (201 Created):

{
  "dId": "LIVING-ROOM-01",
  "versionNo": 1,
  "created": true
}

Channel ingestion (with enrichment)

curl -X POST http://localhost:3000/api/v1/ingest/temperature-readings/garage-sensor \
  -H "Content-Type: application/json" \
  -d '{
    "document": {
      "temperature": 18.2,
      "timestamp": "2026-07-17T10:06:00Z"
    }
  }'

Response (201 Created):

{
  "dId": "GARAGE-TEMP-01",
  "versionNo": 1,
  "created": true
}

TamaDB enriched the document with sensorId and location from the channel bindings before storing.

Update an existing document (upsert)

Send the same sensorId again — TamaDB merges the new temperature into the existing document:

curl -X POST http://localhost:3000/api/v1/ingest/temperature-readings/garage-sensor \
  -H "Content-Type: application/json" \
  -d '{
    "document": {
      "temperature": 19.1,
      "timestamp": "2026-07-17T10:10:00Z"
    }
  }'

Response (200 OK — updated, not created):

{
  "dId": "GARAGE-TEMP-01",
  "versionNo": 2,
  "created": false
}

Step 5: Retrieve Documents

Full document by D-Id

curl http://localhost:3000/api/v1/documents/temperature-readings/GARAGE-TEMP-01

Response:

{
  "sensorId": "GARAGE-TEMP-01",
  "location": "Garage",
  "temperature": 19.1,
  "timestamp": "2026-07-17T10:10:00Z"
}

Retrieve a specific field

curl "http://localhost:3000/api/v1/documents/temperature-readings/GARAGE-TEMP-01?path=temperature"

Response:

{ "_value": 19.1 }

Search by value

curl -X POST http://localhost:3000/api/v1/documents/temperature-readings/search \
  -H "Content-Type: application/json" \
  -d '{ "field": "location", "value": "Garage" }'

Response:

{
  "matches": [
    { "sensorId": "GARAGE-TEMP-01", "location": "Garage", "temperature": 19.1, "timestamp": "2026-07-17T10:10:00Z" }
  ],
  "count": 1
}

Glob pattern matching

# All sensors starting with "GARAGE"
curl http://localhost:3000/api/v1/documents/temperature-readings/GARAGE-*

Step 6: Subscribe to Updates

Register a webhook to receive updates in real-time:

curl -X POST http://localhost:3000/api/v1/subscriptions \
  -H "Content-Type: application/json" \
  -d '{
    "scope": "document_type",
    "documentType": "temperature-readings",
    "callbackUrl": "https://my-dashboard.example.com/webhook/temperatures",
    "options": {
      "deliveryMode": "retry",
      "payloadFormat": "envelope"
    }
  }'

Response (201):

{
  "id": "sub-abc123-0001",
  "scope": "document_type",
  "documentType": "temperature-readings",
  "callbackUrl": "https://my-dashboard.example.com/webhook/temperatures",
  "options": { "deliveryMode": "retry", "payloadFormat": "envelope" },
  "active": true,
  "createdAt": "2026-07-17T10:15:00Z"
}

Every time a temperature reading is stored or updated, TamaDB POSTs to your callback:

{
  "subscriptionId": "sub-abc123-0001",
  "dId": "GARAGE-TEMP-01",
  "documentType": "temperature-readings",
  "document": {
    "sensorId": "GARAGE-TEMP-01",
    "location": "Garage",
    "temperature": 19.1
  },
  "timestamp": "2026-07-17T10:10:00Z"
}

Subscribe to a specific document

curl -X POST http://localhost:3000/api/v1/subscriptions \
  -H "Content-Type: application/json" \
  -d '{
    "scope": "document_id",
    "documentType": "temperature-readings",
    "dId": "LIVING-ROOM-01",
    "callbackUrl": "https://alerts.example.com/living-room"
  }'

Cancel a subscription

curl -X DELETE http://localhost:3000/api/v1/subscriptions/sub-abc123-0001

Step 7: Delete a Document

Soft-delete permanently excludes the document from retrieval and delivery:

curl -X DELETE http://localhost:3000/api/v1/documents/temperature-readings/LIVING-ROOM-01

Response:

{ "deleted": true }

The document's data and history are preserved in storage for audit, but it will never be served again.

Step 8: Optimistic Concurrency

Prevent accidental overwrites by supplying the expected version:

curl -X POST http://localhost:3000/api/v1/ingest/temperature-readings \
  -H "Content-Type: application/json" \
  -d '{
    "document": { "sensorId": "LIVING-ROOM-01", "location": "Living Room", "temperature": 23 },
    "versionNo": 2
  }'

If the document's current version is not 2, TamaDB rejects with:

{
  "code": "VERSION_CONFLICT",
  "message": "Version conflict: expected 2, current is 3"
}

Omit versionNo for a "blind write" (last-write-wins).

Configuration at Runtime

All configuration is changeable without restart via the API.

Log Level

# Check current
curl http://localhost:3000/api/v1/config/log-level

# Change to warn
curl -X PUT http://localhost:3000/api/v1/config/log-level \
  -H "Content-Type: application/json" \
  -d '{ "level": "warn" }'

Document TTL (Instance-level)

curl -X PUT http://localhost:3000/api/v1/config/ttl/instance \
  -H "Content-Type: application/json" \
  -d '{ "ttl": 7200 }'

Storage Strategy

curl http://localhost:3000/api/v1/config/storage-strategy
curl -X PUT http://localhost:3000/api/v1/config/storage-strategy \
  -H "Content-Type: application/json" \
  -d '{ "mode": "memory" }'

Quarantine Policy

curl -X PUT http://localhost:3000/api/v1/config/quarantine-policy \
  -H "Content-Type: application/json" \
  -d '{ "policy": "all-rejections" }'

Enrichment Policies

curl -X PUT http://localhost:3000/api/v1/config/enrichment-policies \
  -H "Content-Type: application/json" \
  -d '{
    "precedence": "environment-over-content",
    "conflictResolution": "accept-and-warn"
  }'

Troubleshooting

Reading Logs

TamaDB outputs structured JSON logs to stdout. Each log entry includes:

• level: debug, info, warn, error
• time: ISO timestamp
• msg: human-readable description
• Contextual fields: dId, documentType, channelId, error details

Example log entries:

{"level":"info","time":"2026-07-17T10:05:00Z","msg":"Document stored","documentType":"temperature-readings","dId":"LIVING-ROOM-01","versionNo":1}
{"level":"warn","time":"2026-07-17T10:06:00Z","msg":"Enrichment conflict resolved","fieldPath":"location","winner":"environment"}
{"level":"error","time":"2026-07-17T10:07:00Z","msg":"Schema validation failed","documentType":"temperature-readings","errors":"Field 'temperature' expected type number"}

Inspecting Quarantined Payloads

When a payload is rejected and the quarantine policy permits it, TamaDB stores the original payload for inspection.

Check the error response for the quarantine reference:

{
  "code": "SCHEMA_VALIDATION_FAILED",
  "message": "Field 'temperature' expected type number, got string",
  "quarantineRef": "qr-m1abc-0001"
}

Retrieve the quarantined payload:

curl http://localhost:3000/api/v1/quarantine/qr-m1abc-0001

List all quarantined entries:

# All entries
curl http://localhost:3000/api/v1/quarantine

# Filter by type
curl "http://localhost:3000/api/v1/quarantine?rejectionType=parse-error"

# Paginate
curl "http://localhost:3000/api/v1/quarantine?limit=10&offset=0"

Common Error Codes

Channel Status Issues

If a channel stops accepting ingestion:

# Check channel status
curl http://localhost:3000/api/v1/config/channels?documentType=temperature-readings

Look at the status field:

• "active" — accepting ingestion (normal)
• "inactive" — operator-suspended. Reactivate: POST /config/channels/{id}/activate
• "blocked" — schema conflict. Fix the Channel_Schema, then activation becomes possible

Composed D-Ids (Multi-Field Identity)

When a schema has multiple identity fields (e.g., customer + order), D-Ids are composed:

{
  "properties": {
    "customerId": { "type": "string", "identityKind": "parent-key" },
    "orderId": { "type": "string", "identityKind": "primary-key" }
  }
}

In URLs, composed D-Ids use : as delimiter:

curl http://localhost:3000/api/v1/documents/orders/customer-A:order-123

In request bodies (store response), they're arrays:

{ "dId": ["customer-A", "order-123"] }

Quick Reference: Key Endpoints

Document version: 1.0 — July 2026. Corresponds to: TamaDB Increment 1

About Me

I'm a software engineer passionate about building reliable, high-performance distributed systems. My focus areas include event-driven architectures and data pipelines.

Background

With experience in designing and implementing transportation-layer systems, I specialise in building purpose-built middleware that sits between data producers and consumers — ensuring data quality, reliability, and real-time delivery.

Current Projects

• TamaDB — A high-performance transportation cache for validated, versioned document delivery
• PamaDB — A planned data publishing layer for transformations and frozen document snapshots

PamaDB

PamaDB is the planned data publishing layer in the TamaDB ecosystem. While TamaDB handles transportation (receive, validate, cache, deliver), PamaDB will handle transformation and publication.

Planned Capabilities

• Data Transformations — Apply rules to transform documents before publishing to downstream systems
• Frozen Documents — Create immutable snapshots of document state for audit and compliance
• Hard Subscriptions — Guaranteed delivery contracts with persistence and retry semantics beyond TamaDB's soft webhooks
• Retrieval Schemas — Shape document output per consumer needs without affecting the stored canonical form

Relationship to TamaDB

PamaDB consumes from TamaDB's delivery pipeline. TamaDB remains the dumb-pipe transportation layer; PamaDB adds intelligence for publishing. They are complementary, not competing.

Contact

Feel free to reach out for questions, collaboration, or feedback on TamaDB and related projects.

Get in Touch

• Email: Available on request
• GitLab: tamagitlab.duckdns.org/teneris

External Links

Code & Repositories

• TamaDB on GitLab — Source code, issues, CI/CD pipeline
• GitHub Profile — Open-source contributions and experiments

Technical Blog

• Technical Blog — Articles on system design, middleware, and data architecture