BigDataKRH API Documentation

BigDataKRH is a unified data pipeline platform. Connect any system using Push, Pull, or Stream — with a single consistent API.

Base URL: /bigdata/api/v1 Format: JSON Auth: API Key

Overview

All API endpoints accept and return JSON. Use HTTPS in production. The platform supports three data transport modes:

PUSH

Push

Your external system sends events to BigDataKRH. Suitable for event-driven architectures, webhooks, and real-time event ingestion.

PULL

Pull

BigDataKRH periodically fetches data from your system. Suitable for polling APIs, scheduled syncs, and batch data ingestion.

STREAM

Stream

Persistent Server-Sent Events connection for real-time data delivery. Suitable for dashboards, live feeds, and 24/7 monitoring.

Authentication

All requests (except /health) require an API key passed in the X-API-Key header.

Request Header

X-API-Key: bd_your_api_key_here

Getting an API key: Register a data source in the Dashboard to receive a unique API key for that source.

API Key Format

All API keys follow the format bd_ followed by 48 hex characters. Example:

Example Key

bd_a3f2c1d4e5b6a7f8c9d0e1f2a3b4c5d6e7f8a9b0c1d2e3f4a5b6c7d8e9f0a1b2

Quick Start

Push your first event in under 2 minutes.

Step 1 — Register a source

POST /api/v1/sources

curl -X POST /bigdata/api/v1/sources \
  -H "Content-Type: application/json" \
  -H "X-API-Key: bd_admin_key_here" \
  -d '{
    "name": "My ERP System",
    "type": "push",
    "description": "Sales events from ERP"
  }'

201 Created

{
  "id": "src_abc123",
  "name": "My ERP System",
  "api_key": "bd_a3f2c1d4...",
  "type": "push",
  "status": "active",
  "created_at": "2026-07-29T10:00:00Z"
}

Step 2 — Push an event

POST /api/v1/push

curl -X POST /bigdata/api/v1/push \
  -H "Content-Type: application/json" \
  -H "X-API-Key: bd_a3f2c1d4..." \
  -d '{
    "source_id": "src_abc123",
    "event": "sale.completed",
    "data": {
      "order_id": 1001,
      "amount": 299.99,
      "currency": "MYR"
    }
  }'

200 OK

{
  "status": "accepted",
  "event_id": "evt_xyz789",
  "timestamp": "2026-07-29T10:00:01Z"
}

POST

/api/v1/push

PUSH

Ingest one or more events from an external system into BigDataKRH. Supports single events and batch arrays.

Request Body

Field	Type	Required	Description
`source_id`	string	Yes	Your registered source ID
`event`	string	Yes	Event type name (e.g. `user.login`)
`data`	object	Yes	Event payload (any valid JSON object)
`timestamp`	string	No	ISO 8601 timestamp. Defaults to server time.
`meta`	object	No	Optional metadata (tags, version, trace_id)

Single Event

Single Push

POST /bigdata/api/v1/push
Content-Type: application/json
X-API-Key: bd_your_key

{
  "source_id": "src_abc123",
  "event": "user.login",
  "data": {
    "user_id": 42,
    "ip": "192.168.1.1",
    "user_agent": "Mozilla/5.0"
  },
  "timestamp": "2026-07-29T10:00:00Z",
  "meta": {
    "version": "2.0",
    "env": "production"
  }
}

Batch Push

Send multiple events in a single request by wrapping them in an events array.

Batch Push

POST /bigdata/api/v1/push
Content-Type: application/json
X-API-Key: bd_your_key

{
  "source_id": "src_abc123",
  "events": [
    { "event": "page.view", "data": { "path": "/home" } },
    { "event": "page.view", "data": { "path": "/products" } },
    { "event": "cart.add",  "data": { "product_id": 99, "qty": 1 } }
  ]
}

200 OK

{
  "status": "accepted",
  "accepted": 3,
  "rejected": 0,
  "event_ids": ["evt_001", "evt_002", "evt_003"]
}

GET

/api/v1/pull

PULL

Query stored events from a source. Supports filtering, pagination, and date ranges.

Query Parameters

Parameter	Type	Default	Description
`source_id`	string	—	Filter by source ID (required)
`event`	string	—	Filter by event type
`from`	ISO 8601	—	Start of date range
`to`	ISO 8601	now	End of date range
`limit`	integer	100	Max records (1–1000)
`offset`	integer	0	Pagination offset
`order`	asc / desc	desc	Sort order by timestamp

Example Pull Request

GET /bigdata/api/v1/pull?source_id=src_abc123&event=user.login&limit=50&from=2026-07-01T00:00:00Z
X-API-Key: bd_your_key

200 OK

{
  "total": 1240,
  "limit": 50,
  "offset": 0,
  "data": [
    {
      "id": "evt_xyz789",
      "source_id": "src_abc123",
      "event": "user.login",
      "data": { "user_id": 42, "ip": "192.168.1.1" },
      "timestamp": "2026-07-29T10:00:00Z"
    }
  ]
}

GET

/api/v1/stream

STREAM

Open a Server-Sent Events (SSE) connection for real-time event delivery. The connection stays open and events are pushed as they arrive, 24/7.

Query Parameters

Parameter	Type	Required	Description
`source_id`	string	Yes	Source to subscribe to
`event`	string	No	Filter by event type
`last_event_id`	string	No	Resume from this event ID

The response uses Content-Type: text/event-stream. The connection stays alive indefinitely. Your client should reconnect automatically on disconnect.

JavaScript Example

EventSource — JavaScript

const url = '/bigdata/api/v1/stream?source_id=src_abc123';
const headers = { 'X-API-Key': 'bd_your_key' };

// Use fetch for SSE with custom headers
const response = await fetch(url, { headers });
const reader = response.body.getReader();
const decoder = new TextDecoder();

while (true) {
  const { value, done } = await reader.read();
  if (done) break;
  const text = decoder.decode(value);
  // Parse SSE format: "data: {...}\n\n"
  text.split('\n\n').forEach(chunk => {
    if (chunk.startsWith('data: ')) {
      const event = JSON.parse(chunk.slice(6));
      console.log('New event:', event);
    }
  });
}

SSE Event Format

Stream Output

id: evt_xyz789
event: user.login
data: {"id":"evt_xyz789","source_id":"src_abc123","event":"user.login","data":{"user_id":42},"timestamp":"2026-07-29T10:00:00Z"}

id: evt_xyz790
event: sale.completed
data: {"id":"evt_xyz790","source_id":"src_abc123","event":"sale.completed","data":{"order_id":1001,"amount":299.99},"timestamp":"2026-07-29T10:00:05Z"}

Data Sources

Manage registered data sources — the origin points of your data.

GET /api/v1/sources List all sources

POST /api/v1/sources Register new source

GET /api/v1/sources/{id} Get source details

PUT /api/v1/sources/{id} Update source config

DELETE /api/v1/sources/{id} Deactivate source

Source Object

Field	Type	Description
`id`	string	Unique source identifier (`src_*`)
`name`	string	Human-readable source name
`type`	push / pull / stream	Transport mode
`status`	active / paused / error	Current source status
`api_key`	string	Source API key (shown once on creation)
`schema`	object	Optional JSON schema for validation
`pull_url`	string	URL to fetch from (pull sources only)
`pull_interval`	integer	Poll interval in seconds (pull sources)
`webhook_url`	string	Webhook to notify on new data
`created_at`	ISO 8601	Creation timestamp

GET

/api/v1/health

Public endpoint — no authentication required. Returns platform health status.

200 OK

{
  "status": "healthy",
  "version": "1.0.0",
  "timestamp": "2026-07-29T10:00:00Z",
  "subsystems": {
    "database": "healthy",
    "queue":    "healthy",
    "storage":  "healthy"
  }
}

Error Handling

All errors return a JSON body with a machine-readable code and human-readable message.

HTTP Status	Code	Description
400	`invalid_request`	Missing or malformed request body
401	`unauthorized`	Missing or invalid API key
403	`forbidden`	API key lacks permission
404	`not_found`	Source or resource not found
422	`schema_error`	Data failed schema validation
429	`rate_limited`	Too many requests
500	`server_error`	Internal server error

401 Unauthorized

{
  "error": {
    "code": "unauthorized",
    "message": "Missing or invalid X-API-Key header"
  }
}

Rate Limits

Rate limits are applied per API key. Headers are returned on every response.

Endpoint	Limit	Window
POST /push	1,000 requests	per minute
GET /pull	300 requests	per minute
GET /stream	10 connections	per key
All others	60 requests	per minute

Rate Limit Headers

X-RateLimit-Limit: 1000
X-RateLimit-Remaining: 987
X-RateLimit-Reset: 1722247260