API docs by Redocly

Event Ingestion API (1.0.0)

Download OpenAPI specification:Download

DataKitchen Support: support@datakitchen.io URL: https://datakitchen.io License: Proprietary, All rights reserved.

Event Ingestion API client for DataKitchen’s DataOps Observability

Events

DatasetOperation Event

Event reports that a read or write operation has occurred for the specified dataset.

Authorizations:
SAKey
header Parameters
EVENT-SOURCE
string
Default: API
Enum: "USER" "SCHEDULER" "API" "RULES_ENGINE"

Set the source of the event. If unset, the Event Ingestion API will assume the source of the Event is API. Warning - This parameter is not intended for use by end users.

Request Body schema: application/json

Data describing the event.

server_name
string or null [ 1 .. 255 ] characters
Default: null

Optional. Human readable display name for the server.

dataset_key
string or null [ 1 .. 255 ] characters
Default: null

The key identifier of the target dataset component for the event action. Only one component key can be provided at a time.

pipeline_name
string or null [ 1 .. 255 ] characters
Default: null

Optional. Human readable display name for the batch-pipeline component.

path
string or null [ 1 .. 4096 ] characters
Default: null

Optional. Path within the dataset where the operation took place.

operation
required
string
Enum: "READ" "WRITE"

Required. The read or write operation performed.

run_key
string or null [ 1 .. 255 ] characters
Default: null

The identifier of the target run for the event action. This key is created and managed by the user. Required if the target component is a batch-pipeline.

payload_keys
Array of strings[ items [ 1 .. 255 ] characters ]

The key identifiers of the datums of interest.

event_timestamp
string <date-time>

Optional. An ISO8601 timestamp that describes when the event occurred. If no timezone is specified, UTC is assumed. If unset, the Event Ingestion API applies its current time as the value.

component_tool
string or null [ 1 .. 255 ] characters ^(?!_)[\w ]*(?<!_)$
Default: null
metadata
object
Default: {}

Optional. Additional key-value information for the event. Provided by the user as needed.

stream_name
string or null [ 1 .. 255 ] characters
Default: null

Optional. Human readable display name for the streaming-pipeline.

external_url
string or null <url>
Default: null

A link to source information.

stream_key
string or null [ 1 .. 255 ] characters
Default: null

The key identifier of the target streaming-pipeline for the event action. Only one component key can be provided at a time.

server_key
string or null [ 1 .. 255 ] characters
Default: null

The key identifier of the target server component for the event action. Only one component key can be provided at a time.

task_name
string or null [ 1 .. 255 ] characters
Default: null

Optional. A human-readable display name for the task.

dataset_name
string or null [ 1 .. 255 ] characters
Default: null

Optional. Human readable display name for the dataset component.

pipeline_key
string or null [ 1 .. 255 ] characters
Default: null

The key identifier of the target batch-pipeline component for the event action. Only one component key can be provided at a time.

task_key
string or null [ 1 .. 255 ] characters
Default: null

Optional. The identifier for a run task. When no task_key is included, the event applies to the run specified by the run_key.

run_name
string or null [ 1 .. 1024 ] characters
Default: null

Optional. Human readable display name for the run.

Responses

Request samples

Content type
application/json
{
  • "server_name": "Azure server",
  • "dataset_key": "897279df-9d37-4e82-8b14-2be4a6a2d9ab",
  • "pipeline_name": "UI Test Pipeline",
  • "path": "path/to/file",
  • "operation": "READ",
  • "run_key": "Important",
  • "payload_keys": "[dataset-11, dataset-59]",
  • "event_timestamp": "2019-08-24T14:15:22Z",
  • "component_tool": null,
  • "metadata": {
    },
  • "stream_name": "UI Test Streaming Pipeline",
  • "external_url": "https://example.com",
  • "stream_key": "897279df-9d37-4e82-8b14-2be4a6a2d9ab",
  • "server_key": "897279df-9d37-4e82-8b14-2be4a6a2d9ab",
  • "task_name": "Descriptive Name",
  • "dataset_name": "Weather data",
  • "pipeline_key": "897279df-9d37-4e82-8b14-2be4a6a2d9ab",
  • "task_key": "18c469e7-db4c-45d4-9414-7c58f196e199",
  • "run_name": "run 2023-02-15"
}

Response samples

Content type
application/json
{
  • "error": "An error message.",
  • "details": {
    }
}

MessageLog Event

Logs a string message related to the pipeline run, and optionally related to a specific task. Post a MessageLog event to capture error, warning, or debugging messages from external tools and scripts.

Authorizations:
SAKey
header Parameters
EVENT-SOURCE
string
Default: API
Enum: "USER" "SCHEDULER" "API" "RULES_ENGINE"

Set the source of the event. If unset, the Event Ingestion API will assume the source of the Event is API. Warning - This parameter is not intended for use by end users.

Request Body schema: application/json

Data describing the event.

server_name
string or null [ 1 .. 255 ] characters
Default: null

Optional. Human readable display name for the server.

log_level
required
string
Enum: "ERROR" "WARNING" "INFO"

Required. The severity level of the message.

dataset_key
string or null [ 1 .. 255 ] characters
Default: null

The key identifier of the target dataset component for the event action. Only one component key can be provided at a time.

pipeline_name
string or null [ 1 .. 255 ] characters
Default: null

Optional. Human readable display name for the batch-pipeline component.

run_key
string or null [ 1 .. 255 ] characters
Default: null

The identifier of the target run for the event action. This key is created and managed by the user. Required if the target component is a batch-pipeline.

message
required
string non-empty

Required. The body of the message to log.

payload_keys
Array of strings[ items [ 1 .. 255 ] characters ]

The key identifiers of the datums of interest.

event_timestamp
string <date-time>

Optional. An ISO8601 timestamp that describes when the event occurred. If no timezone is specified, UTC is assumed. If unset, the Event Ingestion API applies its current time as the value.

component_tool
string or null [ 1 .. 255 ] characters ^(?!_)[\w ]*(?<!_)$
Default: null
metadata
object
Default: {}

Optional. Additional key-value information for the event. Provided by the user as needed.

stream_name
string or null [ 1 .. 255 ] characters
Default: null

Optional. Human readable display name for the streaming-pipeline.

external_url
string or null <url>
Default: null

A link to source information.

stream_key
string or null [ 1 .. 255 ] characters
Default: null

The key identifier of the target streaming-pipeline for the event action. Only one component key can be provided at a time.

server_key
string or null [ 1 .. 255 ] characters
Default: null

The key identifier of the target server component for the event action. Only one component key can be provided at a time.

task_name
string or null [ 1 .. 255 ] characters
Default: null

Optional. A human-readable display name for the task.

dataset_name
string or null [ 1 .. 255 ] characters
Default: null

Optional. Human readable display name for the dataset component.

pipeline_key
string or null [ 1 .. 255 ] characters
Default: null

The key identifier of the target batch-pipeline component for the event action. Only one component key can be provided at a time.

task_key
string or null [ 1 .. 255 ] characters
Default: null

Optional. The identifier for a run task. When no task_key is included, the event applies to the run specified by the run_key.

run_name
string or null [ 1 .. 1024 ] characters
Default: null

Optional. Human readable display name for the run.

Responses

Request samples

Content type
application/json
{
  • "server_name": "Azure server",
  • "log_level": "ERROR",
  • "dataset_key": "897279df-9d37-4e82-8b14-2be4a6a2d9ab",
  • "pipeline_name": "UI Test Pipeline",
  • "run_key": "Important",
  • "message": "The job has completed.",
  • "payload_keys": "[dataset-11, dataset-59]",
  • "event_timestamp": "2019-08-24T14:15:22Z",
  • "component_tool": null,
  • "metadata": {
    },
  • "stream_name": "UI Test Streaming Pipeline",
  • "external_url": "https://example.com",
  • "stream_key": "897279df-9d37-4e82-8b14-2be4a6a2d9ab",
  • "server_key": "897279df-9d37-4e82-8b14-2be4a6a2d9ab",
  • "task_name": "Descriptive Name",
  • "dataset_name": "Weather data",
  • "pipeline_key": "897279df-9d37-4e82-8b14-2be4a6a2d9ab",
  • "task_key": "18c469e7-db4c-45d4-9414-7c58f196e199",
  • "run_name": "run 2023-02-15"
}

Response samples

Content type
application/json
{
  • "error": "An error message.",
  • "details": {
    }
}

MetricLog Event

Logs the value of a user-defined datum of interest, such as a row count. Post a MetricLog event to track the value of a metric through a run or for comparing the value of the metric across multiple runs.

Authorizations:
SAKey
header Parameters
EVENT-SOURCE
string
Default: API
Enum: "USER" "SCHEDULER" "API" "RULES_ENGINE"

Set the source of the event. If unset, the Event Ingestion API will assume the source of the Event is API. Warning - This parameter is not intended for use by end users.

Request Body schema: application/json

Data describing the event.

server_name
string or null [ 1 .. 255 ] characters
Default: null

Optional. Human readable display name for the server.

dataset_key
string or null [ 1 .. 255 ] characters
Default: null

The key identifier of the target dataset component for the event action. Only one component key can be provided at a time.

pipeline_name
string or null [ 1 .. 255 ] characters
Default: null

Optional. Human readable display name for the batch-pipeline component.

run_key
string or null [ 1 .. 255 ] characters
Default: null

The identifier of the target run for the event action. This key is created and managed by the user. Required if the target component is a batch-pipeline.

payload_keys
Array of strings[ items [ 1 .. 255 ] characters ]

The key identifiers of the datums of interest.

metric_key
required
string non-empty

Required. A string to identify the metric_value data source. Used to group like metrics. This value is created and managed by the user.

event_timestamp
string <date-time>

Optional. An ISO8601 timestamp that describes when the event occurred. If no timezone is specified, UTC is assumed. If unset, the Event Ingestion API applies its current time as the value.

component_tool
string or null [ 1 .. 255 ] characters ^(?!_)[\w ]*(?<!_)$
Default: null
metadata
object
Default: {}

Optional. Additional key-value information for the event. Provided by the user as needed.

stream_name
string or null [ 1 .. 255 ] characters
Default: null

Optional. Human readable display name for the streaming-pipeline.

external_url
string or null <url>
Default: null

A link to source information.

stream_key
string or null [ 1 .. 255 ] characters
Default: null

The key identifier of the target streaming-pipeline for the event action. Only one component key can be provided at a time.

server_key
string or null [ 1 .. 255 ] characters
Default: null

The key identifier of the target server component for the event action. Only one component key can be provided at a time.

metric_value
required
number

Required. The data value to be logged. Decimal numerals only; NaN/INF values not supported.

task_name
string or null [ 1 .. 255 ] characters
Default: null

Optional. A human-readable display name for the task.

dataset_name
string or null [ 1 .. 255 ] characters
Default: null

Optional. Human readable display name for the dataset component.

pipeline_key
string or null [ 1 .. 255 ] characters
Default: null

The key identifier of the target batch-pipeline component for the event action. Only one component key can be provided at a time.

task_key
string or null [ 1 .. 255 ] characters
Default: null

Optional. The identifier for a run task. When no task_key is included, the event applies to the run specified by the run_key.

run_name
string or null [ 1 .. 1024 ] characters
Default: null

Optional. Human readable display name for the run.

Responses

Request samples

Content type
application/json
{
  • "server_name": "Azure server",
  • "dataset_key": "897279df-9d37-4e82-8b14-2be4a6a2d9ab",
  • "pipeline_name": "UI Test Pipeline",
  • "run_key": "Important",
  • "payload_keys": "[dataset-11, dataset-59]",
  • "metric_key": "aaff79f0-5e10-4038-b847-28c0e5f42f0d",
  • "event_timestamp": "2019-08-24T14:15:22Z",
  • "component_tool": null,
  • "metadata": {
    },
  • "stream_name": "UI Test Streaming Pipeline",
  • "external_url": "https://example.com",
  • "stream_key": "897279df-9d37-4e82-8b14-2be4a6a2d9ab",
  • "server_key": "897279df-9d37-4e82-8b14-2be4a6a2d9ab",
  • "metric_value": 10.22,
  • "task_name": "Descriptive Name",
  • "dataset_name": "Weather data",
  • "pipeline_key": "897279df-9d37-4e82-8b14-2be4a6a2d9ab",
  • "task_key": "18c469e7-db4c-45d4-9414-7c58f196e199",
  • "run_name": "run 2023-02-15"
}

Response samples

Content type
application/json
{
  • "error": "An error message.",
  • "details": {
    }
}

RunStatus Event

Changes the status of a specified batch pipeline run or of a specified task in the run. Post a RunStatus event to alert the system that a run has started and ended with a certain status or that a task has started and ended with a certain status.

Authorizations:
SAKey
header Parameters
EVENT-SOURCE
string
Default: API
Enum: "USER" "SCHEDULER" "API" "RULES_ENGINE"

Set the source of the event. If unset, the Event Ingestion API will assume the source of the Event is API. Warning - This parameter is not intended for use by end users.

Request Body schema: application/json

Data describing the event.

metadata
object
Default: {}

Optional. Additional key-value information for the event. Provided by the user as needed.

stream_name
string or null [ 1 .. 255 ] characters
Default: null

Optional. Human readable display name for the streaming-pipeline.

server_name
string or null [ 1 .. 255 ] characters
Default: null

Optional. Human readable display name for the server.

payload_keys
Array of strings[ items [ 1 .. 255 ] characters ]

The key identifiers of the datums of interest.

task_name
string or null [ 1 .. 255 ] characters
Default: null

Optional. A human-readable display name for the task.

dataset_name
string or null [ 1 .. 255 ] characters
Default: null

Optional. Human readable display name for the dataset component.

status
required
string
Enum: "RUNNING" "COMPLETED" "COMPLETED_WITH_WARNINGS" "FAILED"

Required. The status to be applied. Can set the status for both runs and tasks. A run starts when no task_key is present and status is “RUNNING” and ends when no task_key is present and status is not running.

external_url
string or null <url>
Default: null

A link to source information.

stream_key
string or null [ 1 .. 255 ] characters
Default: null

The key identifier of the target streaming-pipeline for the event action. Only one component key can be provided at a time.

pipeline_key
string or null [ 1 .. 255 ] characters
Default: null

The key identifier of the target batch-pipeline component for the event action. Only one component key can be provided at a time.

dataset_key
string or null [ 1 .. 255 ] characters
Default: null

The key identifier of the target dataset component for the event action. Only one component key can be provided at a time.

task_key
string or null [ 1 .. 255 ] characters
Default: null

Optional. The identifier for a run task. When no task_key is included, the event applies to the run specified by the run_key.

pipeline_name
string or null [ 1 .. 255 ] characters
Default: null

Optional. Human readable display name for the batch-pipeline component.

event_timestamp
string <date-time>

Optional. An ISO8601 timestamp that describes when the event occurred. If no timezone is specified, UTC is assumed. If unset, the Event Ingestion API applies its current time as the value.

server_key
string or null [ 1 .. 255 ] characters
Default: null

The key identifier of the target server component for the event action. Only one component key can be provided at a time.

run_key
string or null [ 1 .. 255 ] characters
Default: null

The identifier of the target run for the event action. This key is created and managed by the user. Required if the target component is a batch-pipeline.

component_tool
string or null [ 1 .. 255 ] characters ^(?!_)[\w ]*(?<!_)$
Default: null
run_name
string or null [ 1 .. 1024 ] characters
Default: null

Optional. Human readable display name for the run.

Responses

Request samples

Content type
application/json
{
  • "metadata": {
    },
  • "stream_name": "UI Test Streaming Pipeline",
  • "server_name": "Azure server",
  • "payload_keys": "[dataset-11, dataset-59]",
  • "task_name": "Descriptive Name",
  • "dataset_name": "Weather data",
  • "status": "RUNNING",
  • "external_url": "https://example.com",
  • "stream_key": "897279df-9d37-4e82-8b14-2be4a6a2d9ab",
  • "pipeline_key": "897279df-9d37-4e82-8b14-2be4a6a2d9ab",
  • "dataset_key": "897279df-9d37-4e82-8b14-2be4a6a2d9ab",
  • "task_key": "18c469e7-db4c-45d4-9414-7c58f196e199",
  • "pipeline_name": "UI Test Pipeline",
  • "event_timestamp": "2019-08-24T14:15:22Z",
  • "server_key": "897279df-9d37-4e82-8b14-2be4a6a2d9ab",
  • "run_key": "Important",
  • "component_tool": null,
  • "run_name": "run 2023-02-15"
}

Response samples

Content type
application/json
{
  • "error": "An error message.",
  • "details": {
    }
}

TestOutcomes Event

Reports the outcomes of a test or a set of tests. Receives a list of test outcomes by test name and status (PASSED, FAILED, or WARNING). Post a TestOutcomesEvent event to send outcomes from an external tool to the Observe platform to track and report on runs and outcomes.

Authorizations:
SAKey
header Parameters
EVENT-SOURCE
string
Default: API
Enum: "USER" "SCHEDULER" "API" "RULES_ENGINE"

Set the source of the event. If unset, the Event Ingestion API will assume the source of the Event is API. Warning - This parameter is not intended for use by end users.

Request Body schema: application/json

Data describing the event.

server_name
string or null [ 1 .. 255 ] characters
Default: null

Optional. Human readable display name for the server.

dataset_key
string or null [ 1 .. 255 ] characters
Default: null

The key identifier of the target dataset component for the event action. Only one component key can be provided at a time.

pipeline_name
string or null [ 1 .. 255 ] characters
Default: null

Optional. Human readable display name for the batch-pipeline component.

required
Array of objects (TestOutcomeItem) [ 1 .. 500 ] items

Required. A list of objects, each representing the outcomes of a test.

run_key
string or null [ 1 .. 255 ] characters
Default: null

The identifier of the target run for the event action. This key is created and managed by the user. Required if the target component is a batch-pipeline.

payload_keys
Array of strings[ items [ 1 .. 255 ] characters ]

The key identifiers of the datums of interest.

event_timestamp
string <date-time>

Optional. An ISO8601 timestamp that describes when the event occurred. If no timezone is specified, UTC is assumed. If unset, the Event Ingestion API applies its current time as the value.

component_tool
string or null [ 1 .. 255 ] characters ^(?!_)[\w ]*(?<!_)$
Default: null
metadata
object
Default: {}

Optional. Additional key-value information for the event. Provided by the user as needed.

object or null

Optional. Test Outcomes additional component data.

stream_name
string or null [ 1 .. 255 ] characters
Default: null

Optional. Human readable display name for the streaming-pipeline.

external_url
string or null <url>
Default: null

A link to source information.

stream_key
string or null [ 1 .. 255 ] characters
Default: null

The key identifier of the target streaming-pipeline for the event action. Only one component key can be provided at a time.

server_key
string or null [ 1 .. 255 ] characters
Default: null

The key identifier of the target server component for the event action. Only one component key can be provided at a time.

task_name
string or null [ 1 .. 255 ] characters
Default: null

Optional. A human-readable display name for the task.

dataset_name
string or null [ 1 .. 255 ] characters
Default: null

Optional. Human readable display name for the dataset component.

pipeline_key
string or null [ 1 .. 255 ] characters
Default: null

The key identifier of the target batch-pipeline component for the event action. Only one component key can be provided at a time.

task_key
string or null [ 1 .. 255 ] characters
Default: null

Optional. The identifier for a run task. When no task_key is included, the event applies to the run specified by the run_key.

run_name
string or null [ 1 .. 1024 ] characters
Default: null

Optional. Human readable display name for the run.

Responses

Request samples

Content type
application/json
{
  • "server_name": "Azure server",
  • "dataset_key": "897279df-9d37-4e82-8b14-2be4a6a2d9ab",
  • "pipeline_name": "UI Test Pipeline",
  • "test_outcomes": [
    ],
  • "run_key": "Important",
  • "payload_keys": "[dataset-11, dataset-59]",
  • "event_timestamp": "2019-08-24T14:15:22Z",
  • "component_tool": null,
  • "metadata": {
    },
  • "component_integrations": {
    },
  • "stream_name": "UI Test Streaming Pipeline",
  • "external_url": "https://example.com",
  • "stream_key": "897279df-9d37-4e82-8b14-2be4a6a2d9ab",
  • "server_key": "897279df-9d37-4e82-8b14-2be4a6a2d9ab",
  • "task_name": "Descriptive Name",
  • "dataset_name": "Weather data",
  • "pipeline_key": "897279df-9d37-4e82-8b14-2be4a6a2d9ab",
  • "task_key": "18c469e7-db4c-45d4-9414-7c58f196e199",
  • "run_name": "run 2023-02-15"
}

Response samples

Content type
application/json
{
  • "error": "An error message.",
  • "details": {
    }
}