API docs by Redocly

Event Ingestion API (1.0.0)

Download OpenAPI specification:Download

DataKitchen Support: support@datakitchen.io URL: https://datakitchen.io License: Proprietary, All rights reserved.

Event Ingestion API client for DataKitchen’s DataOps Observability

Events

DatasetOperation Event

Event reports that a read or write operation has occurred for the specified dataset.

Authorizations:
SAKey
header Parameters
EVENT-SOURCE
string
Default: API
Enum: "USER" "SCHEDULER" "API" "RULES_ENGINE"

Set the source of the event. If unset, the Event Ingestion API will assume the source of the Event is API. Warning - This parameter is not intended for use by end users.

Request Body schema: application/json

Data describing the event.

task_key
string or null [ 1 .. 255 ] characters
Default: null

Optional. The identifier for a run task. When no task_key is included, the event applies to the run specified by the run_key.

stream_name
string or null [ 1 .. 255 ] characters
Default: null

Optional. Human readable display name for the streaming-pipeline.

run_key
string or null [ 1 .. 255 ] characters
Default: null

The identifier of the target run for the event action. This key is created and managed by the user. Required if the target component is a batch-pipeline.

task_name
string or null [ 1 .. 255 ] characters
Default: null

Optional. A human-readable display name for the task.

stream_key
string or null [ 1 .. 255 ] characters
Default: null

The key identifier of the target streaming-pipeline for the event action. Only one component key can be provided at a time.

dataset_key
string or null [ 1 .. 255 ] characters
Default: null

The key identifier of the target dataset component for the event action. Only one component key can be provided at a time.

operation
required
string
Enum: "READ" "WRITE"

Required. The read or write operation performed.

component_tool
string or null [ 1 .. 255 ] characters ^(?!_)[\w ]*(?<!_)$
Default: null
external_url
string or null <url>
Default: null

A link to source information.

event_timestamp
string <date-time>

Optional. An ISO8601 timestamp that describes when the event occurred. If no timezone is specified, UTC is assumed. If unset, the Event Ingestion API applies its current time as the value.

metadata
object
Default: {}

Optional. Additional key-value information for the event. Provided by the user as needed.

run_name
string or null [ 1 .. 1024 ] characters
Default: null

Optional. Human readable display name for the run.

path
string or null [ 1 .. 4096 ] characters
Default: null

Optional. Path within the dataset where the operation took place.

pipeline_key
string or null [ 1 .. 255 ] characters
Default: null

The key identifier of the target batch-pipeline component for the event action. Only one component key can be provided at a time.

payload_keys
Array of strings[ items [ 1 .. 255 ] characters ]

The key identifiers of the datums of interest.

server_key
string or null [ 1 .. 255 ] characters
Default: null

The key identifier of the target server component for the event action. Only one component key can be provided at a time.

server_name
string or null [ 1 .. 255 ] characters
Default: null

Optional. Human readable display name for the server.

dataset_name
string or null [ 1 .. 255 ] characters
Default: null

Optional. Human readable display name for the dataset component.

pipeline_name
string or null [ 1 .. 255 ] characters
Default: null

Optional. Human readable display name for the batch-pipeline component.

Responses

Request samples

Content type
application/json
{
  • "task_key": "18c469e7-db4c-45d4-9414-7c58f196e199",
  • "stream_name": "UI Test Streaming Pipeline",
  • "run_key": "Important",
  • "task_name": "Descriptive Name",
  • "stream_key": "897279df-9d37-4e82-8b14-2be4a6a2d9ab",
  • "dataset_key": "897279df-9d37-4e82-8b14-2be4a6a2d9ab",
  • "operation": "READ",
  • "component_tool": null,
  • "external_url": "https://example.com",
  • "event_timestamp": "2019-08-24T14:15:22Z",
  • "metadata": {
    },
  • "run_name": "run 2023-02-15",
  • "path": "path/to/file",
  • "pipeline_key": "897279df-9d37-4e82-8b14-2be4a6a2d9ab",
  • "payload_keys": "[dataset-11, dataset-59]",
  • "server_key": "897279df-9d37-4e82-8b14-2be4a6a2d9ab",
  • "server_name": "Azure server",
  • "dataset_name": "Weather data",
  • "pipeline_name": "UI Test Pipeline"
}

Response samples

Content type
application/json
{
  • "error": "An error message.",
  • "details": {
    }
}

MessageLog Event

Logs a string message related to the pipeline run, and optionally related to a specific task. Post a MessageLog event to capture error, warning, or debugging messages from external tools and scripts.

Authorizations:
SAKey
header Parameters
EVENT-SOURCE
string
Default: API
Enum: "USER" "SCHEDULER" "API" "RULES_ENGINE"

Set the source of the event. If unset, the Event Ingestion API will assume the source of the Event is API. Warning - This parameter is not intended for use by end users.

Request Body schema: application/json

Data describing the event.

task_key
string or null [ 1 .. 255 ] characters
Default: null

Optional. The identifier for a run task. When no task_key is included, the event applies to the run specified by the run_key.

stream_name
string or null [ 1 .. 255 ] characters
Default: null

Optional. Human readable display name for the streaming-pipeline.

run_key
string or null [ 1 .. 255 ] characters
Default: null

The identifier of the target run for the event action. This key is created and managed by the user. Required if the target component is a batch-pipeline.

task_name
string or null [ 1 .. 255 ] characters
Default: null

Optional. A human-readable display name for the task.

stream_key
string or null [ 1 .. 255 ] characters
Default: null

The key identifier of the target streaming-pipeline for the event action. Only one component key can be provided at a time.

dataset_key
string or null [ 1 .. 255 ] characters
Default: null

The key identifier of the target dataset component for the event action. Only one component key can be provided at a time.

component_tool
string or null [ 1 .. 255 ] characters ^(?!_)[\w ]*(?<!_)$
Default: null
external_url
string or null <url>
Default: null

A link to source information.

event_timestamp
string <date-time>

Optional. An ISO8601 timestamp that describes when the event occurred. If no timezone is specified, UTC is assumed. If unset, the Event Ingestion API applies its current time as the value.

metadata
object
Default: {}

Optional. Additional key-value information for the event. Provided by the user as needed.

run_name
string or null [ 1 .. 1024 ] characters
Default: null

Optional. Human readable display name for the run.

pipeline_key
string or null [ 1 .. 255 ] characters
Default: null

The key identifier of the target batch-pipeline component for the event action. Only one component key can be provided at a time.

message
required
string non-empty

Required. The body of the message to log.

log_level
required
string
Enum: "ERROR" "WARNING" "INFO"

Required. The severity level of the message.

payload_keys
Array of strings[ items [ 1 .. 255 ] characters ]

The key identifiers of the datums of interest.

server_key
string or null [ 1 .. 255 ] characters
Default: null

The key identifier of the target server component for the event action. Only one component key can be provided at a time.

server_name
string or null [ 1 .. 255 ] characters
Default: null

Optional. Human readable display name for the server.

dataset_name
string or null [ 1 .. 255 ] characters
Default: null

Optional. Human readable display name for the dataset component.

pipeline_name
string or null [ 1 .. 255 ] characters
Default: null

Optional. Human readable display name for the batch-pipeline component.

Responses

Request samples

Content type
application/json
{
  • "task_key": "18c469e7-db4c-45d4-9414-7c58f196e199",
  • "stream_name": "UI Test Streaming Pipeline",
  • "run_key": "Important",
  • "task_name": "Descriptive Name",
  • "stream_key": "897279df-9d37-4e82-8b14-2be4a6a2d9ab",
  • "dataset_key": "897279df-9d37-4e82-8b14-2be4a6a2d9ab",
  • "component_tool": null,
  • "external_url": "https://example.com",
  • "event_timestamp": "2019-08-24T14:15:22Z",
  • "metadata": {
    },
  • "run_name": "run 2023-02-15",
  • "pipeline_key": "897279df-9d37-4e82-8b14-2be4a6a2d9ab",
  • "message": "The job has completed.",
  • "log_level": "ERROR",
  • "payload_keys": "[dataset-11, dataset-59]",
  • "server_key": "897279df-9d37-4e82-8b14-2be4a6a2d9ab",
  • "server_name": "Azure server",
  • "dataset_name": "Weather data",
  • "pipeline_name": "UI Test Pipeline"
}

Response samples

Content type
application/json
{
  • "error": "An error message.",
  • "details": {
    }
}

MetricLog Event

Logs the value of a user-defined datum of interest, such as a row count. Post a MetricLog event to track the value of a metric through a run or for comparing the value of the metric across multiple runs.

Authorizations:
SAKey
header Parameters
EVENT-SOURCE
string
Default: API
Enum: "USER" "SCHEDULER" "API" "RULES_ENGINE"

Set the source of the event. If unset, the Event Ingestion API will assume the source of the Event is API. Warning - This parameter is not intended for use by end users.

Request Body schema: application/json

Data describing the event.

task_key
string or null [ 1 .. 255 ] characters
Default: null

Optional. The identifier for a run task. When no task_key is included, the event applies to the run specified by the run_key.

stream_name
string or null [ 1 .. 255 ] characters
Default: null

Optional. Human readable display name for the streaming-pipeline.

run_key
string or null [ 1 .. 255 ] characters
Default: null

The identifier of the target run for the event action. This key is created and managed by the user. Required if the target component is a batch-pipeline.

task_name
string or null [ 1 .. 255 ] characters
Default: null

Optional. A human-readable display name for the task.

stream_key
string or null [ 1 .. 255 ] characters
Default: null

The key identifier of the target streaming-pipeline for the event action. Only one component key can be provided at a time.

dataset_key
string or null [ 1 .. 255 ] characters
Default: null

The key identifier of the target dataset component for the event action. Only one component key can be provided at a time.

metric_key
required
string non-empty

Required. A string to identify the metric_value data source. Used to group like metrics. This value is created and managed by the user.

component_tool
string or null [ 1 .. 255 ] characters ^(?!_)[\w ]*(?<!_)$
Default: null
external_url
string or null <url>
Default: null

A link to source information.

event_timestamp
string <date-time>

Optional. An ISO8601 timestamp that describes when the event occurred. If no timezone is specified, UTC is assumed. If unset, the Event Ingestion API applies its current time as the value.

metadata
object
Default: {}

Optional. Additional key-value information for the event. Provided by the user as needed.

run_name
string or null [ 1 .. 1024 ] characters
Default: null

Optional. Human readable display name for the run.

pipeline_key
string or null [ 1 .. 255 ] characters
Default: null

The key identifier of the target batch-pipeline component for the event action. Only one component key can be provided at a time.

payload_keys
Array of strings[ items [ 1 .. 255 ] characters ]

The key identifiers of the datums of interest.

server_key
string or null [ 1 .. 255 ] characters
Default: null

The key identifier of the target server component for the event action. Only one component key can be provided at a time.

server_name
string or null [ 1 .. 255 ] characters
Default: null

Optional. Human readable display name for the server.

dataset_name
string or null [ 1 .. 255 ] characters
Default: null

Optional. Human readable display name for the dataset component.

pipeline_name
string or null [ 1 .. 255 ] characters
Default: null

Optional. Human readable display name for the batch-pipeline component.

metric_value
required
number

Required. The data value to be logged. Decimal numerals only; NaN/INF values not supported.

Responses

Request samples

Content type
application/json
{
  • "task_key": "18c469e7-db4c-45d4-9414-7c58f196e199",
  • "stream_name": "UI Test Streaming Pipeline",
  • "run_key": "Important",
  • "task_name": "Descriptive Name",
  • "stream_key": "897279df-9d37-4e82-8b14-2be4a6a2d9ab",
  • "dataset_key": "897279df-9d37-4e82-8b14-2be4a6a2d9ab",
  • "metric_key": "aaff79f0-5e10-4038-b847-28c0e5f42f0d",
  • "component_tool": null,
  • "external_url": "https://example.com",
  • "event_timestamp": "2019-08-24T14:15:22Z",
  • "metadata": {
    },
  • "run_name": "run 2023-02-15",
  • "pipeline_key": "897279df-9d37-4e82-8b14-2be4a6a2d9ab",
  • "payload_keys": "[dataset-11, dataset-59]",
  • "server_key": "897279df-9d37-4e82-8b14-2be4a6a2d9ab",
  • "server_name": "Azure server",
  • "dataset_name": "Weather data",
  • "pipeline_name": "UI Test Pipeline",
  • "metric_value": 10.22
}

Response samples

Content type
application/json
{
  • "error": "An error message.",
  • "details": {
    }
}

RunStatus Event

Changes the status of a specified batch pipeline run or of a specified task in the run. Post a RunStatus event to alert the system that a run has started and ended with a certain status or that a task has started and ended with a certain status.

Authorizations:
SAKey
header Parameters
EVENT-SOURCE
string
Default: API
Enum: "USER" "SCHEDULER" "API" "RULES_ENGINE"

Set the source of the event. If unset, the Event Ingestion API will assume the source of the Event is API. Warning - This parameter is not intended for use by end users.

Request Body schema: application/json

Data describing the event.

server_key
string or null [ 1 .. 255 ] characters
Default: null

The key identifier of the target server component for the event action. Only one component key can be provided at a time.

server_name
string or null [ 1 .. 255 ] characters
Default: null

Optional. Human readable display name for the server.

task_key
string or null [ 1 .. 255 ] characters
Default: null

Optional. The identifier for a run task. When no task_key is included, the event applies to the run specified by the run_key.

dataset_name
string or null [ 1 .. 255 ] characters
Default: null

Optional. Human readable display name for the dataset component.

status
required
string
Enum: "RUNNING" "COMPLETED" "COMPLETED_WITH_WARNINGS" "FAILED"

Required. The status to be applied. Can set the status for both runs and tasks. A run starts when no task_key is present and status is “RUNNING” and ends when no task_key is present and status is not running.

stream_name
string or null [ 1 .. 255 ] characters
Default: null

Optional. Human readable display name for the streaming-pipeline.

pipeline_name
string or null [ 1 .. 255 ] characters
Default: null

Optional. Human readable display name for the batch-pipeline component.

run_key
string or null [ 1 .. 255 ] characters
Default: null

The identifier of the target run for the event action. This key is created and managed by the user. Required if the target component is a batch-pipeline.

event_timestamp
string <date-time>

Optional. An ISO8601 timestamp that describes when the event occurred. If no timezone is specified, UTC is assumed. If unset, the Event Ingestion API applies its current time as the value.

metadata
object
Default: {}

Optional. Additional key-value information for the event. Provided by the user as needed.

task_name
string or null [ 1 .. 255 ] characters
Default: null

Optional. A human-readable display name for the task.

run_name
string or null [ 1 .. 1024 ] characters
Default: null

Optional. Human readable display name for the run.

stream_key
string or null [ 1 .. 255 ] characters
Default: null

The key identifier of the target streaming-pipeline for the event action. Only one component key can be provided at a time.

dataset_key
string or null [ 1 .. 255 ] characters
Default: null

The key identifier of the target dataset component for the event action. Only one component key can be provided at a time.

payload_keys
Array of strings[ items [ 1 .. 255 ] characters ]

The key identifiers of the datums of interest.

component_tool
string or null [ 1 .. 255 ] characters ^(?!_)[\w ]*(?<!_)$
Default: null
pipeline_key
string or null [ 1 .. 255 ] characters
Default: null

The key identifier of the target batch-pipeline component for the event action. Only one component key can be provided at a time.

external_url
string or null <url>
Default: null

A link to source information.

Responses

Request samples

Content type
application/json
{
  • "server_key": "897279df-9d37-4e82-8b14-2be4a6a2d9ab",
  • "server_name": "Azure server",
  • "task_key": "18c469e7-db4c-45d4-9414-7c58f196e199",
  • "dataset_name": "Weather data",
  • "status": "RUNNING",
  • "stream_name": "UI Test Streaming Pipeline",
  • "pipeline_name": "UI Test Pipeline",
  • "run_key": "Important",
  • "event_timestamp": "2019-08-24T14:15:22Z",
  • "metadata": {
    },
  • "task_name": "Descriptive Name",
  • "run_name": "run 2023-02-15",
  • "stream_key": "897279df-9d37-4e82-8b14-2be4a6a2d9ab",
  • "dataset_key": "897279df-9d37-4e82-8b14-2be4a6a2d9ab",
  • "payload_keys": "[dataset-11, dataset-59]",
  • "component_tool": null,
  • "pipeline_key": "897279df-9d37-4e82-8b14-2be4a6a2d9ab",
  • "external_url": "https://example.com"
}

Response samples

Content type
application/json
{
  • "error": "An error message.",
  • "details": {
    }
}

TestOutcomes Event

Reports the outcomes of a test or a set of tests. Receives a list of test outcomes by test name and status (PASSED, FAILED, or WARNING). Post a TestOutcomesEvent event to send outcomes from an external tool to the Observe platform to track and report on runs and outcomes.

Authorizations:
SAKey
header Parameters
EVENT-SOURCE
string
Default: API
Enum: "USER" "SCHEDULER" "API" "RULES_ENGINE"

Set the source of the event. If unset, the Event Ingestion API will assume the source of the Event is API. Warning - This parameter is not intended for use by end users.

Request Body schema: application/json

Data describing the event.

task_key
string or null [ 1 .. 255 ] characters
Default: null

Optional. The identifier for a run task. When no task_key is included, the event applies to the run specified by the run_key.

stream_name
string or null [ 1 .. 255 ] characters
Default: null

Optional. Human readable display name for the streaming-pipeline.

run_key
string or null [ 1 .. 255 ] characters
Default: null

The identifier of the target run for the event action. This key is created and managed by the user. Required if the target component is a batch-pipeline.

task_name
string or null [ 1 .. 255 ] characters
Default: null

Optional. A human-readable display name for the task.

stream_key
string or null [ 1 .. 255 ] characters
Default: null

The key identifier of the target streaming-pipeline for the event action. Only one component key can be provided at a time.

dataset_key
string or null [ 1 .. 255 ] characters
Default: null

The key identifier of the target dataset component for the event action. Only one component key can be provided at a time.

component_tool
string or null [ 1 .. 255 ] characters ^(?!_)[\w ]*(?<!_)$
Default: null
external_url
string or null <url>
Default: null

A link to source information.

event_timestamp
string <date-time>

Optional. An ISO8601 timestamp that describes when the event occurred. If no timezone is specified, UTC is assumed. If unset, the Event Ingestion API applies its current time as the value.

metadata
object
Default: {}

Optional. Additional key-value information for the event. Provided by the user as needed.

object or null

Optional. Test Outcomes additional component data.

run_name
string or null [ 1 .. 1024 ] characters
Default: null

Optional. Human readable display name for the run.

pipeline_key
string or null [ 1 .. 255 ] characters
Default: null

The key identifier of the target batch-pipeline component for the event action. Only one component key can be provided at a time.

required
Array of objects (TestOutcomeItem) [ 1 .. 500 ] items

Required. A list of objects, each representing the outcomes of a test.

payload_keys
Array of strings[ items [ 1 .. 255 ] characters ]

The key identifiers of the datums of interest.

server_key
string or null [ 1 .. 255 ] characters
Default: null

The key identifier of the target server component for the event action. Only one component key can be provided at a time.

server_name
string or null [ 1 .. 255 ] characters
Default: null

Optional. Human readable display name for the server.

dataset_name
string or null [ 1 .. 255 ] characters
Default: null

Optional. Human readable display name for the dataset component.

pipeline_name
string or null [ 1 .. 255 ] characters
Default: null

Optional. Human readable display name for the batch-pipeline component.

Responses

Request samples

Content type
application/json
{
  • "task_key": "18c469e7-db4c-45d4-9414-7c58f196e199",
  • "stream_name": "UI Test Streaming Pipeline",
  • "run_key": "Important",
  • "task_name": "Descriptive Name",
  • "stream_key": "897279df-9d37-4e82-8b14-2be4a6a2d9ab",
  • "dataset_key": "897279df-9d37-4e82-8b14-2be4a6a2d9ab",
  • "component_tool": null,
  • "external_url": "https://example.com",
  • "event_timestamp": "2019-08-24T14:15:22Z",
  • "metadata": {
    },
  • "component_integrations": {
    },
  • "run_name": "run 2023-02-15",
  • "pipeline_key": "897279df-9d37-4e82-8b14-2be4a6a2d9ab",
  • "test_outcomes": [
    ],
  • "payload_keys": "[dataset-11, dataset-59]",
  • "server_key": "897279df-9d37-4e82-8b14-2be4a6a2d9ab",
  • "server_name": "Azure server",
  • "dataset_name": "Weather data",
  • "pipeline_name": "UI Test Pipeline"
}

Response samples

Content type
application/json
{
  • "error": "An error message.",
  • "details": {
    }
}