API docs by Redocly

Event Ingestion API (1.0.0)

Download OpenAPI specification:Download

DataKitchen Support: support@datakitchen.io URL: https://datakitchen.io License: Proprietary, All rights reserved.

Event Ingestion API client for DataKitchen’s DataOps Observability

Events

DatasetOperation Event

Event reports that a read or write operation has occurred for the specified dataset.

Authorizations:
SAKey
header Parameters
EVENT-SOURCE
string
Default: API
Enum: "USER" "SCHEDULER" "API" "RULES_ENGINE"

Set the source of the event. If unset, the Event Ingestion API will assume the source of the Event is API. Warning - This parameter is not intended for use by end users.

Request Body schema: application/json

Data describing the event.

run_name
string or null [ 1 .. 1024 ] characters
Default: null

Optional. Human readable display name for the run.

component_tool
string or null [ 1 .. 255 ] characters ^(?!_)[\w ]*(?<!_)$
Default: null
stream_name
string or null [ 1 .. 255 ] characters
Default: null

Optional. Human readable display name for the streaming-pipeline.

server_key
string or null [ 1 .. 255 ] characters
Default: null

The key identifier of the target server component for the event action. Only one component key can be provided at a time.

task_key
string or null [ 1 .. 255 ] characters
Default: null

Optional. The identifier for a run task. When no task_key is included, the event applies to the run specified by the run_key.

task_name
string or null [ 1 .. 255 ] characters
Default: null

Optional. A human-readable display name for the task.

dataset_name
string or null [ 1 .. 255 ] characters
Default: null

Optional. Human readable display name for the dataset component.

server_name
string or null [ 1 .. 255 ] characters
Default: null

Optional. Human readable display name for the server.

pipeline_name
string or null [ 1 .. 255 ] characters
Default: null

Optional. Human readable display name for the batch-pipeline component.

payload_keys
Array of strings[ items [ 1 .. 255 ] characters ]

The key identifiers of the datums of interest.

metadata
object
Default: {}

Optional. Additional key-value information for the event. Provided by the user as needed.

run_key
string or null [ 1 .. 255 ] characters
Default: null

The identifier of the target run for the event action. This key is created and managed by the user. Required if the target component is a batch-pipeline.

pipeline_key
string or null [ 1 .. 255 ] characters
Default: null

The key identifier of the target batch-pipeline component for the event action. Only one component key can be provided at a time.

event_timestamp
string <date-time>

Optional. An ISO8601 timestamp that describes when the event occurred. If no timezone is specified, UTC is assumed. If unset, the Event Ingestion API applies its current time as the value.

operation
required
string
Enum: "READ" "WRITE"

Required. The read or write operation performed.

dataset_key
string or null [ 1 .. 255 ] characters
Default: null

The key identifier of the target dataset component for the event action. Only one component key can be provided at a time.

external_url
string or null <url>
Default: null

A link to source information.

stream_key
string or null [ 1 .. 255 ] characters
Default: null

The key identifier of the target streaming-pipeline for the event action. Only one component key can be provided at a time.

path
string or null [ 1 .. 4096 ] characters
Default: null

Optional. Path within the dataset where the operation took place.

Responses

Request samples

Content type
application/json
{
  • "run_name": "run 2023-02-15",
  • "component_tool": null,
  • "stream_name": "UI Test Streaming Pipeline",
  • "server_key": "897279df-9d37-4e82-8b14-2be4a6a2d9ab",
  • "task_key": "18c469e7-db4c-45d4-9414-7c58f196e199",
  • "task_name": "Descriptive Name",
  • "dataset_name": "Weather data",
  • "server_name": "Azure server",
  • "pipeline_name": "UI Test Pipeline",
  • "payload_keys": "[dataset-11, dataset-59]",
  • "metadata": {
    },
  • "run_key": "Important",
  • "pipeline_key": "897279df-9d37-4e82-8b14-2be4a6a2d9ab",
  • "event_timestamp": "2019-08-24T14:15:22Z",
  • "operation": "READ",
  • "dataset_key": "897279df-9d37-4e82-8b14-2be4a6a2d9ab",
  • "external_url": "https://example.com",
  • "stream_key": "897279df-9d37-4e82-8b14-2be4a6a2d9ab",
  • "path": "path/to/file"
}

Response samples

Content type
application/json
{
  • "error": "An error message.",
  • "details": {
    }
}

MessageLog Event

Logs a string message related to the pipeline run, and optionally related to a specific task. Post a MessageLog event to capture error, warning, or debugging messages from external tools and scripts.

Authorizations:
SAKey
header Parameters
EVENT-SOURCE
string
Default: API
Enum: "USER" "SCHEDULER" "API" "RULES_ENGINE"

Set the source of the event. If unset, the Event Ingestion API will assume the source of the Event is API. Warning - This parameter is not intended for use by end users.

Request Body schema: application/json

Data describing the event.

run_name
string or null [ 1 .. 1024 ] characters
Default: null

Optional. Human readable display name for the run.

message
required
string non-empty

Required. The body of the message to log.

component_tool
string or null [ 1 .. 255 ] characters ^(?!_)[\w ]*(?<!_)$
Default: null
stream_name
string or null [ 1 .. 255 ] characters
Default: null

Optional. Human readable display name for the streaming-pipeline.

server_key
string or null [ 1 .. 255 ] characters
Default: null

The key identifier of the target server component for the event action. Only one component key can be provided at a time.

task_key
string or null [ 1 .. 255 ] characters
Default: null

Optional. The identifier for a run task. When no task_key is included, the event applies to the run specified by the run_key.

task_name
string or null [ 1 .. 255 ] characters
Default: null

Optional. A human-readable display name for the task.

dataset_name
string or null [ 1 .. 255 ] characters
Default: null

Optional. Human readable display name for the dataset component.

server_name
string or null [ 1 .. 255 ] characters
Default: null

Optional. Human readable display name for the server.

pipeline_name
string or null [ 1 .. 255 ] characters
Default: null

Optional. Human readable display name for the batch-pipeline component.

payload_keys
Array of strings[ items [ 1 .. 255 ] characters ]

The key identifiers of the datums of interest.

metadata
object
Default: {}

Optional. Additional key-value information for the event. Provided by the user as needed.

run_key
string or null [ 1 .. 255 ] characters
Default: null

The identifier of the target run for the event action. This key is created and managed by the user. Required if the target component is a batch-pipeline.

pipeline_key
string or null [ 1 .. 255 ] characters
Default: null

The key identifier of the target batch-pipeline component for the event action. Only one component key can be provided at a time.

event_timestamp
string <date-time>

Optional. An ISO8601 timestamp that describes when the event occurred. If no timezone is specified, UTC is assumed. If unset, the Event Ingestion API applies its current time as the value.

dataset_key
string or null [ 1 .. 255 ] characters
Default: null

The key identifier of the target dataset component for the event action. Only one component key can be provided at a time.

external_url
string or null <url>
Default: null

A link to source information.

stream_key
string or null [ 1 .. 255 ] characters
Default: null

The key identifier of the target streaming-pipeline for the event action. Only one component key can be provided at a time.

log_level
required
string
Enum: "ERROR" "WARNING" "INFO"

Required. The severity level of the message.

Responses

Request samples

Content type
application/json
{
  • "run_name": "run 2023-02-15",
  • "message": "The job has completed.",
  • "component_tool": null,
  • "stream_name": "UI Test Streaming Pipeline",
  • "server_key": "897279df-9d37-4e82-8b14-2be4a6a2d9ab",
  • "task_key": "18c469e7-db4c-45d4-9414-7c58f196e199",
  • "task_name": "Descriptive Name",
  • "dataset_name": "Weather data",
  • "server_name": "Azure server",
  • "pipeline_name": "UI Test Pipeline",
  • "payload_keys": "[dataset-11, dataset-59]",
  • "metadata": {
    },
  • "run_key": "Important",
  • "pipeline_key": "897279df-9d37-4e82-8b14-2be4a6a2d9ab",
  • "event_timestamp": "2019-08-24T14:15:22Z",
  • "dataset_key": "897279df-9d37-4e82-8b14-2be4a6a2d9ab",
  • "external_url": "https://example.com",
  • "stream_key": "897279df-9d37-4e82-8b14-2be4a6a2d9ab",
  • "log_level": "ERROR"
}

Response samples

Content type
application/json
{
  • "error": "An error message.",
  • "details": {
    }
}

MetricLog Event

Logs the value of a user-defined datum of interest, such as a row count. Post a MetricLog event to track the value of a metric through a run or for comparing the value of the metric across multiple runs.

Authorizations:
SAKey
header Parameters
EVENT-SOURCE
string
Default: API
Enum: "USER" "SCHEDULER" "API" "RULES_ENGINE"

Set the source of the event. If unset, the Event Ingestion API will assume the source of the Event is API. Warning - This parameter is not intended for use by end users.

Request Body schema: application/json

Data describing the event.

run_name
string or null [ 1 .. 1024 ] characters
Default: null

Optional. Human readable display name for the run.

component_tool
string or null [ 1 .. 255 ] characters ^(?!_)[\w ]*(?<!_)$
Default: null
stream_name
string or null [ 1 .. 255 ] characters
Default: null

Optional. Human readable display name for the streaming-pipeline.

server_key
string or null [ 1 .. 255 ] characters
Default: null

The key identifier of the target server component for the event action. Only one component key can be provided at a time.

task_key
string or null [ 1 .. 255 ] characters
Default: null

Optional. The identifier for a run task. When no task_key is included, the event applies to the run specified by the run_key.

task_name
string or null [ 1 .. 255 ] characters
Default: null

Optional. A human-readable display name for the task.

dataset_name
string or null [ 1 .. 255 ] characters
Default: null

Optional. Human readable display name for the dataset component.

server_name
string or null [ 1 .. 255 ] characters
Default: null

Optional. Human readable display name for the server.

pipeline_name
string or null [ 1 .. 255 ] characters
Default: null

Optional. Human readable display name for the batch-pipeline component.

metric_value
required
number

Required. The data value to be logged. Decimal numerals only; NaN/INF values not supported.

payload_keys
Array of strings[ items [ 1 .. 255 ] characters ]

The key identifiers of the datums of interest.

metadata
object
Default: {}

Optional. Additional key-value information for the event. Provided by the user as needed.

run_key
string or null [ 1 .. 255 ] characters
Default: null

The identifier of the target run for the event action. This key is created and managed by the user. Required if the target component is a batch-pipeline.

pipeline_key
string or null [ 1 .. 255 ] characters
Default: null

The key identifier of the target batch-pipeline component for the event action. Only one component key can be provided at a time.

event_timestamp
string <date-time>

Optional. An ISO8601 timestamp that describes when the event occurred. If no timezone is specified, UTC is assumed. If unset, the Event Ingestion API applies its current time as the value.

dataset_key
string or null [ 1 .. 255 ] characters
Default: null

The key identifier of the target dataset component for the event action. Only one component key can be provided at a time.

external_url
string or null <url>
Default: null

A link to source information.

stream_key
string or null [ 1 .. 255 ] characters
Default: null

The key identifier of the target streaming-pipeline for the event action. Only one component key can be provided at a time.

metric_key
required
string non-empty

Required. A string to identify the metric_value data source. Used to group like metrics. This value is created and managed by the user.

Responses

Request samples

Content type
application/json
{
  • "run_name": "run 2023-02-15",
  • "component_tool": null,
  • "stream_name": "UI Test Streaming Pipeline",
  • "server_key": "897279df-9d37-4e82-8b14-2be4a6a2d9ab",
  • "task_key": "18c469e7-db4c-45d4-9414-7c58f196e199",
  • "task_name": "Descriptive Name",
  • "dataset_name": "Weather data",
  • "server_name": "Azure server",
  • "pipeline_name": "UI Test Pipeline",
  • "metric_value": 10.22,
  • "payload_keys": "[dataset-11, dataset-59]",
  • "metadata": {
    },
  • "run_key": "Important",
  • "pipeline_key": "897279df-9d37-4e82-8b14-2be4a6a2d9ab",
  • "event_timestamp": "2019-08-24T14:15:22Z",
  • "dataset_key": "897279df-9d37-4e82-8b14-2be4a6a2d9ab",
  • "external_url": "https://example.com",
  • "stream_key": "897279df-9d37-4e82-8b14-2be4a6a2d9ab",
  • "metric_key": "aaff79f0-5e10-4038-b847-28c0e5f42f0d"
}

Response samples

Content type
application/json
{
  • "error": "An error message.",
  • "details": {
    }
}

RunStatus Event

Changes the status of a specified batch pipeline run or of a specified task in the run. Post a RunStatus event to alert the system that a run has started and ended with a certain status or that a task has started and ended with a certain status.

Authorizations:
SAKey
header Parameters
EVENT-SOURCE
string
Default: API
Enum: "USER" "SCHEDULER" "API" "RULES_ENGINE"

Set the source of the event. If unset, the Event Ingestion API will assume the source of the Event is API. Warning - This parameter is not intended for use by end users.

Request Body schema: application/json

Data describing the event.

stream_name
string or null [ 1 .. 255 ] characters
Default: null

Optional. Human readable display name for the streaming-pipeline.

payload_keys
Array of strings[ items [ 1 .. 255 ] characters ]

The key identifiers of the datums of interest.

dataset_key
string or null [ 1 .. 255 ] characters
Default: null

The key identifier of the target dataset component for the event action. Only one component key can be provided at a time.

server_key
string or null [ 1 .. 255 ] characters
Default: null

The key identifier of the target server component for the event action. Only one component key can be provided at a time.

external_url
string or null <url>
Default: null

A link to source information.

run_name
string or null [ 1 .. 1024 ] characters
Default: null

Optional. Human readable display name for the run.

stream_key
string or null [ 1 .. 255 ] characters
Default: null

The key identifier of the target streaming-pipeline for the event action. Only one component key can be provided at a time.

task_key
string or null [ 1 .. 255 ] characters
Default: null

Optional. The identifier for a run task. When no task_key is included, the event applies to the run specified by the run_key.

task_name
string or null [ 1 .. 255 ] characters
Default: null

Optional. A human-readable display name for the task.

dataset_name
string or null [ 1 .. 255 ] characters
Default: null

Optional. Human readable display name for the dataset component.

server_name
string or null [ 1 .. 255 ] characters
Default: null

Optional. Human readable display name for the server.

metadata
object
Default: {}

Optional. Additional key-value information for the event. Provided by the user as needed.

pipeline_name
string or null [ 1 .. 255 ] characters
Default: null

Optional. Human readable display name for the batch-pipeline component.

run_key
string or null [ 1 .. 255 ] characters
Default: null

The identifier of the target run for the event action. This key is created and managed by the user. Required if the target component is a batch-pipeline.

pipeline_key
string or null [ 1 .. 255 ] characters
Default: null

The key identifier of the target batch-pipeline component for the event action. Only one component key can be provided at a time.

event_timestamp
string <date-time>

Optional. An ISO8601 timestamp that describes when the event occurred. If no timezone is specified, UTC is assumed. If unset, the Event Ingestion API applies its current time as the value.

status
required
string
Enum: "RUNNING" "COMPLETED" "COMPLETED_WITH_WARNINGS" "FAILED"

Required. The status to be applied. Can set the status for both runs and tasks. A run starts when no task_key is present and status is “RUNNING” and ends when no task_key is present and status is not running.

component_tool
string or null [ 1 .. 255 ] characters ^(?!_)[\w ]*(?<!_)$
Default: null

Responses

Request samples

Content type
application/json
{
  • "stream_name": "UI Test Streaming Pipeline",
  • "payload_keys": "[dataset-11, dataset-59]",
  • "dataset_key": "897279df-9d37-4e82-8b14-2be4a6a2d9ab",
  • "server_key": "897279df-9d37-4e82-8b14-2be4a6a2d9ab",
  • "external_url": "https://example.com",
  • "run_name": "run 2023-02-15",
  • "stream_key": "897279df-9d37-4e82-8b14-2be4a6a2d9ab",
  • "task_key": "18c469e7-db4c-45d4-9414-7c58f196e199",
  • "task_name": "Descriptive Name",
  • "dataset_name": "Weather data",
  • "server_name": "Azure server",
  • "metadata": {
    },
  • "pipeline_name": "UI Test Pipeline",
  • "run_key": "Important",
  • "pipeline_key": "897279df-9d37-4e82-8b14-2be4a6a2d9ab",
  • "event_timestamp": "2019-08-24T14:15:22Z",
  • "status": "RUNNING",
  • "component_tool": null
}

Response samples

Content type
application/json
{
  • "error": "An error message.",
  • "details": {
    }
}

TestOutcomes Event

Reports the outcomes of a test or a set of tests. Receives a list of test outcomes by test name and status (PASSED, FAILED, or WARNING). Post a TestOutcomesEvent event to send outcomes from an external tool to the Observe platform to track and report on runs and outcomes.

Authorizations:
SAKey
header Parameters
EVENT-SOURCE
string
Default: API
Enum: "USER" "SCHEDULER" "API" "RULES_ENGINE"

Set the source of the event. If unset, the Event Ingestion API will assume the source of the Event is API. Warning - This parameter is not intended for use by end users.

Request Body schema: application/json

Data describing the event.

run_name
string or null [ 1 .. 1024 ] characters
Default: null

Optional. Human readable display name for the run.

component_tool
string or null [ 1 .. 255 ] characters ^(?!_)[\w ]*(?<!_)$
Default: null
stream_name
string or null [ 1 .. 255 ] characters
Default: null

Optional. Human readable display name for the streaming-pipeline.

server_key
string or null [ 1 .. 255 ] characters
Default: null

The key identifier of the target server component for the event action. Only one component key can be provided at a time.

task_key
string or null [ 1 .. 255 ] characters
Default: null

Optional. The identifier for a run task. When no task_key is included, the event applies to the run specified by the run_key.

task_name
string or null [ 1 .. 255 ] characters
Default: null

Optional. A human-readable display name for the task.

dataset_name
string or null [ 1 .. 255 ] characters
Default: null

Optional. Human readable display name for the dataset component.

server_name
string or null [ 1 .. 255 ] characters
Default: null

Optional. Human readable display name for the server.

pipeline_name
string or null [ 1 .. 255 ] characters
Default: null

Optional. Human readable display name for the batch-pipeline component.

object or null

Optional. Test Outcomes additional component data.

payload_keys
Array of strings[ items [ 1 .. 255 ] characters ]

The key identifiers of the datums of interest.

metadata
object
Default: {}

Optional. Additional key-value information for the event. Provided by the user as needed.

required
Array of objects (TestOutcomeItem) [ 1 .. 500 ] items

Required. A list of objects, each representing the outcomes of a test.

run_key
string or null [ 1 .. 255 ] characters
Default: null

The identifier of the target run for the event action. This key is created and managed by the user. Required if the target component is a batch-pipeline.

pipeline_key
string or null [ 1 .. 255 ] characters
Default: null

The key identifier of the target batch-pipeline component for the event action. Only one component key can be provided at a time.

event_timestamp
string <date-time>

Optional. An ISO8601 timestamp that describes when the event occurred. If no timezone is specified, UTC is assumed. If unset, the Event Ingestion API applies its current time as the value.

dataset_key
string or null [ 1 .. 255 ] characters
Default: null

The key identifier of the target dataset component for the event action. Only one component key can be provided at a time.

external_url
string or null <url>
Default: null

A link to source information.

stream_key
string or null [ 1 .. 255 ] characters
Default: null

The key identifier of the target streaming-pipeline for the event action. Only one component key can be provided at a time.

Responses

Request samples

Content type
application/json
{
  • "run_name": "run 2023-02-15",
  • "component_tool": null,
  • "stream_name": "UI Test Streaming Pipeline",
  • "server_key": "897279df-9d37-4e82-8b14-2be4a6a2d9ab",
  • "task_key": "18c469e7-db4c-45d4-9414-7c58f196e199",
  • "task_name": "Descriptive Name",
  • "dataset_name": "Weather data",
  • "server_name": "Azure server",
  • "pipeline_name": "UI Test Pipeline",
  • "component_integrations": {
    },
  • "payload_keys": "[dataset-11, dataset-59]",
  • "metadata": {
    },
  • "test_outcomes": [
    ],
  • "run_key": "Important",
  • "pipeline_key": "897279df-9d37-4e82-8b14-2be4a6a2d9ab",
  • "event_timestamp": "2019-08-24T14:15:22Z",
  • "dataset_key": "897279df-9d37-4e82-8b14-2be4a6a2d9ab",
  • "external_url": "https://example.com",
  • "stream_key": "897279df-9d37-4e82-8b14-2be4a6a2d9ab"
}

Response samples

Content type
application/json
{
  • "error": "An error message.",
  • "details": {
    }
}