job.proto

path mgmt/v1alpha1/job.proto

package mgmt.v1alpha1

---

Messages

ActivityFailure

Name	Type	Description
message	string

ActivityOptions

Config that contains various timeouts that are configured in the underlying temporal workflow(s) and activities

Name	Type	Description
schedule_to_close_timeout	optional int64	Total time that a workflow is willing to wait for an activity to complete, including retries. Measured in seconds
start_to_close_timeout	optional int64	Max time of a single Temporal Activity execution attempt. This timeout should be as short as the longest psosible execution of any activity (e.g. table sync). Important to know that this is per retry attempt. Defaults to the schedule to close timeout if not provided. Measured in seconds
retry_policy	RetryPolicy	Optionally define a retry policy for the activity If max attempts is not set, the activity will retry indefinitely until the start to close timeout lapses

AiGenerateSourceOptions

Name	Type	Description
ai_connection_id	string	The connection id that corresponds with an AI-based Neosync connection
schemas	repeated AiGenerateSourceSchemaOption	The list of schemas (and their tables) along with any configuration options that will be used to generate data for.
fk_source_connection_id	optional string	An optional connection id that will be used as the basis for the shape of data to be generated.
model_name	string	The name of the model to use
user_prompt	optional string	Optionally provide a user prompt to give more context to the schema
generate_batch_size	optional int64	The batch size that will be used to generate X number of records. This is global and will be applied to all tables configured.

AiGenerateSourceSchemaOption

Name	Type	Description
schema	string	The dataabase schema
tables	repeated AiGenerateSourceTableOption	The list of tables (and their configuration) that reside within the schema to receive generated data

AiGenerateSourceTableOption

Name	Type	Description
table	string	The table that will be used to. 1. The schema of the table will be injected into the prompt, of which the resulting data will then be inserted.
row_count	int64	The total number of records to be generated.

AwsS3DestinationConnectionOptions

Name	Type	Description
storage_class	AwsS3DestinationConnectionOptions.StorageClass	The storage class that will be used when objects are written to S3
max_in_flight	optional uint32	The maximum number of batched messages to have in flight at a given time. Increase this to improve throughput.
timeout	optional string	The maximum period (duration string) to wait on an upload before abandoning it and reattempting.
batch	BatchConfig	Configure batching options to more efficiently store records in S3

AwsS3SourceConnectionOptions

Name	Type	Description
connection_id	string

BatchConfig

Name	Type	Description
count	optional uint32	The max allowed in a batch before it is flushed. 0 to disable.
period	optional string	A duration string in which an incomplete batch should be flushed regardless of the count. Examples are 1s, 1m, 500ms

CancelJobRunRequest

Name	Type	Description
job_run_id	string
account_id	string

CancelJobRunResponse

Name	Type	Description

ColumnError

Name	Type	Description
schema	string
table	string
column	string
errors	repeated string

CreateJobDestination

Name	Type	Description
connection_id	string
options	JobDestinationOptions

CreateJobDestinationConnectionsRequest

Name	Type	Description
job_id	string
destinations	repeated CreateJobDestination

CreateJobDestinationConnectionsResponse

Name	Type	Description
job	Job

CreateJobHookRequest

Name	Type	Description
job_id	string	The unique identifier of the job
hook	NewJobHook	The new hook configuration

CreateJobHookResponse

Name	Type	Description
hook	JobHook	The newly created hook

CreateJobRequest

Name	Type	Description
account_id	string	The unique account identifier that this job will be associated with
job_name	string	The unique, friendly name of the job. This is unique per account
cron_schedule	optional string	Optionally provide a cron schedule. Goes into effect if the job status is set to enabled
mappings	repeated JobMapping
source	JobSource
destinations	repeated CreateJobDestination
initiate_job_run	bool	Initially trigger a run of this job regardless of its status or cron schedule
workflow_options	WorkflowOptions	Specify timeouts and other workflow options for the underlying temporal workflow
sync_options	ActivityOptions	Specify timeout and retry options for data synchronization activities Data sync activities are any piece of work that involves actually synchronizing data from a source to a destination For the data sync and generate jobs, this will be applied per table
virtual_foreign_keys	repeated VirtualForeignConstraint

CreateJobResponse

Name	Type	Description
job	Job

CreateJobRunRequest

Name	Type	Description
job_id	string

CreateJobRunResponse

Name	Type	Description

DatabaseError

Name	Type	Description
errors	repeated string

DeleteJobDestinationConnectionRequest

Name	Type	Description
destination_id	string

DeleteJobDestinationConnectionResponse

Name	Type	Description

DeleteJobHookRequest

Name	Type	Description
id	string	The unique identifier of the hook

DeleteJobHookResponse

Name	Type	Description

DeleteJobRequest

Name	Type	Description
id	string

DeleteJobResponse

Name	Type	Description

DeleteJobRunRequest

Name	Type	Description
job_run_id	string
account_id	string

DeleteJobRunResponse

Name	Type	Description

DynamoDBDestinationConnectionOptions

Configuration for DynamoDB Destination Connection Job Options

Name	Type	Description
table_mappings	repeated DynamoDBDestinationTableMapping	List of table mappings when piping data from a dynamoDB table to another dynamoDB table

DynamoDBDestinationTableMapping

Configuration for mapping a source table to a destination table for DynamoDB

Name	Type	Description
source_table	string	The name of the incoming source table
destination_table	string	The name of the outgoing destination table

DynamoDBSourceConnectionOptions

DynamoDB connection options for a job source

Name	Type	Description
connection_id	string	The unique connection id to a dynamodb connection configuration
tables	repeated DynamoDBSourceTableOption	List of table option configurations for any mapped source table. Any table listed in this must also be present as a job mapping table to be applied.
unmapped_transforms	DynamoDBSourceUnmappedTransformConfig	Default transformations for any unmapped keys
enable_consistent_read	bool	Enforces strong read consistency False: Eventually Consistent Reads, True: Strongly Consistent Reads https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/HowItWorks.ReadConsistency.html

DynamoDBSourceSchemaSubset

Name	Type	Description
tables	repeated DynamoDBSourceTableOption

DynamoDBSourceTableOption

Name	Type	Description
table	string	The table that this configuration will be applied to
where_clause	optional string	An optional PartiQL query that may be used for subsetting the DynamoDB table. This is not a parameterized query and must be valid. Intended to be everything after the WHERE keyword.

DynamoDBSourceUnmappedTransformConfig

Name	Type	Description
b	JobMappingTransformer	Byte
boolean	JobMappingTransformer	Boolean
n	JobMappingTransformer	Number
s	JobMappingTransformer	String

GcpCloudStorageDestinationConnectionOptions

Configuration for Google Cloud Storage Destination Connection Job Options

Name	Type	Description

GenerateSourceOptions

Name	Type	Description
schemas	repeated GenerateSourceSchemaOption
fk_source_connection_id	optional string

GenerateSourceSchemaOption

Name	Type	Description
schema	string
tables	repeated GenerateSourceTableOption

GenerateSourceTableOption

Name	Type	Description
table	string
row_count	int64

GetActiveJobHooksByTimingRequest

Name	Type	Description
job_id	string	The unique identifier of the job
timing	GetActiveJobHooksByTimingRequest.Timing	The timing desired.

GetActiveJobHooksByTimingResponse

Name	Type	Description
hooks	repeated JobHook	The active job hooks

GetJobHookRequest

Name	Type	Description
id	string	The unique identifier of the hook

GetJobHookResponse

Name	Type	Description
hook	JobHook	The found hook

GetJobHooksRequest

Name	Type	Description
job_id	string	The unique identifier of the job

GetJobHooksResponse

Name	Type	Description
hooks	repeated JobHook	The list of hooks found attached to the job

GetJobNextRunsRequest

Name	Type	Description
job_id	string

GetJobNextRunsResponse

Name	Type	Description
next_runs	JobNextRuns

GetJobRecentRunsRequest

Name	Type	Description
job_id	string

GetJobRecentRunsResponse

Name	Type	Description
recent_runs	repeated JobRecentRun

GetJobRequest

Name	Type	Description
id	string

GetJobResponse

Name	Type	Description
job	Job

GetJobRunEventsRequest

Name	Type	Description
job_run_id	string
account_id	string

GetJobRunEventsResponse

Name	Type	Description
events	repeated JobRunEvent
is_run_complete	bool

GetJobRunLogsStreamRequest

Name	Type	Description
job_run_id	string
account_id	string
window	LogWindow	The time window in which to retrieve the logs
should_tail	bool	Whether or not to tail the stream. Note: only works with k8s-pods and is not currently supported with Loki logs
max_log_lines	optional int64	Optionally provide a max log limit
log_levels	repeated LogLevel	Provide a list of log levels to filter by. If any of these are UNSPECIFIED, all log levels are returned.

GetJobRunLogsStreamResponse

Name	Type	Description
log_line	string
timestamp	optional google.protobuf.Timestamp

GetJobRunRequest

Name	Type	Description
job_run_id	string
account_id	string

GetJobRunResponse

Name	Type	Description
job_run	JobRun

GetJobRunsRequest

Name	Type	Description
job_id	string
account_id	string

GetJobRunsResponse

Name	Type	Description
job_runs	repeated JobRun

GetJobStatusRequest

Name	Type	Description
job_id	string

GetJobStatusResponse

Name	Type	Description
status	JobStatus

GetJobStatusesRequest

Name	Type	Description
account_id	string

GetJobStatusesResponse

Name	Type	Description
statuses	repeated JobStatusRecord

GetJobsRequest

Name	Type	Description
account_id	string

GetJobsResponse

Name	Type	Description
jobs	repeated Job

GetRunContextRequest

Name	Type	Description
id	RunContextKey

GetRunContextResponse

Name	Type	Description
value	bytes

IsJobHookNameAvailableRequest

Name	Type	Description
job_id	string	The unique identifier of the job
name	string	The hook name to check.

IsJobHookNameAvailableResponse

Name	Type	Description
is_available	bool	Whether or not the hook name is available

IsJobNameAvailableRequest

Name	Type	Description
name	string
account_id	string

IsJobNameAvailableResponse

Name	Type	Description
is_available	bool

Job

Name	Type	Description
id	string	The unique identifier of the job
created_by_user_id	string
created_at	google.protobuf.Timestamp
updated_by_user_id	string
updated_at	google.protobuf.Timestamp
name	string	The unique, friendly name of the job
source	JobSource
destinations	repeated JobDestination
mappings	repeated JobMapping
cron_schedule	optional string
account_id	string	The account identifier that a job is associated with
sync_options	ActivityOptions	Specify timeout and retry options for data synchronization activities Data sync activities are any piece of work that involves actually synchronizing data from a source to a destination For the data sync and generate jobs, this will be applied per table
workflow_options	WorkflowOptions	Specify timeouts and other workflow options for the underlying temporal workflow
virtual_foreign_keys	repeated VirtualForeignConstraint	Any virtual foreign keys that are configured as a part of this job

JobDestination

Name	Type	Description
connection_id	string
options	JobDestinationOptions
id	string

JobDestinationOptions

Name	Type	Description
postgres_options	PostgresDestinationConnectionOptions
aws_s3_options	AwsS3DestinationConnectionOptions
mysql_options	MysqlDestinationConnectionOptions
mongodb_options	MongoDBDestinationConnectionOptions
gcp_cloudstorage_options	GcpCloudStorageDestinationConnectionOptions	Destination Connecton options for Google Cloud Storage
dynamodb_options	DynamoDBDestinationConnectionOptions	Destination Connection options for DynamoDB
mssql_options	MssqlDestinationConnectionOptions	Destination Connection options for Microsoft SQL Server

JobHook

Name	Type	Description
id	string	The unique identifier of this hook.
name	string	Name of the hook for display/reference.
description	string	Description of what this hook does.
job_id	string	The unique identifier of the job this hook belongs to.
config	JobHookConfig	Hook-type specific configuration.
created_by_user_id	string	The user that created this hook.
created_at	google.protobuf.Timestamp	The time this hook was created.
updated_by_user_id	string	The user that last updated this hook.
updated_at	google.protobuf.Timestamp	The last time this hook was updated.
enabled	bool	Whether or not the hook is enabled.
priority	uint32	The priority of the hook (0-100). This determines the execution order. Lower values are higher priority (priority=0 is the highest). Tie Breaking is determined by the following: (priority, created_at, id) in ascending order.

JobHookConfig

Contains the specific hook type configurations.

Name	Type	Description
sql	JobHookConfig.JobSqlHook	Configuration for SQL-specific hooks.

JobHookConfig.JobSqlHook

Configuration for SQL-based hooks

Name	Type	Description
query	string	The SQL query to execute
connection_id	string	Unique identifier of the Neosync connection to run this hook for. Must be a connection id that is present in the job.
timing	JobHookConfig.JobSqlHook.Timing	The timing of when the hook will run

JobHookConfig.JobSqlHook.Timing

Name	Type	Description
pre_sync	JobHookTimingPreSync	A Pre-Sync timing. Will run before the first table sync. Will run before Truncation, if enabled. Will run before Schema Init, if enabled.
post_sync	JobHookTimingPostSync	A Post-Sync timing. Will run after the last table sync.

JobHookTimingPostSync

Configures the job hook to run after the last table sync.

Name	Type	Description

JobHookTimingPreSync

Configures the job hook to run before the first table sync. Will run before Truncation, if enabled. Will run before Schema Init, if enabled.

Name	Type	Description

JobMapping

Name	Type	Description
schema	string
table	string
column	string
transformer	JobMappingTransformer

JobMappingTransformer

Name	Type	Description
source	TransformerSource	@deprecated - This is no longer used in favor just providing the TransformerConfig
config	TransformerConfig

JobNextRuns

Name	Type	Description
next_run_times	repeated google.protobuf.Timestamp

JobRecentRun

Name	Type	Description
start_time	google.protobuf.Timestamp
job_run_id	string

JobRun

Name	Type	Description
id	string	The id of the job run. This will currently be equivalent to the temporal workflow id
job_id	string	The unique identifier of the job id this run is associated with
name	string	The name of the job run.
status	JobRunStatus	the status of the job run
started_at	google.protobuf.Timestamp	A timestamp of when the run started
completed_at	optional google.protobuf.Timestamp	Available if the run completed or has not yet been archived by the system
pending_activities	repeated PendingActivity	Pending activities are only returned when retrieving a specific job run and will not be returned when requesting job runs in list format

JobRunEvent

Name	Type	Description
id	int64
type	string
start_time	google.protobuf.Timestamp
close_time	google.protobuf.Timestamp
metadata	JobRunEventMetadata
tasks	repeated JobRunEventTask

JobRunEventMetadata

Name	Type	Description
sync_metadata	JobRunSyncMetadata

JobRunEventTask

Name	Type	Description
id	int64
type	string
event_time	google.protobuf.Timestamp
error	JobRunEventTaskError

JobRunEventTaskError

Name	Type	Description
message	string
retry_state	string

JobRunSyncMetadata

Name	Type	Description
schema	string
table	string

JobSource

Name	Type	Description
options	JobSourceOptions

JobSourceOptions

Name	Type	Description
postgres	PostgresSourceConnectionOptions
aws_s3	AwsS3SourceConnectionOptions
mysql	MysqlSourceConnectionOptions
generate	GenerateSourceOptions
ai_generate	AiGenerateSourceOptions
mongodb	MongoDBSourceConnectionOptions
dynamodb	DynamoDBSourceConnectionOptions
mssql	MssqlSourceConnectionOptions

JobSourceSqlSubetSchemas

Name	Type	Description
postgres_subset	PostgresSourceSchemaSubset
mysql_subset	MysqlSourceSchemaSubset
dynamodb_subset	DynamoDBSourceSchemaSubset
mssql_subset	MssqlSourceSchemaSubset

JobStatusRecord

Name	Type	Description
job_id	string
status	JobStatus

MongoDBDestinationConnectionOptions

Name	Type	Description

MongoDBSourceConnectionOptions

MongoDB connection options for a job source

Name	Type	Description
connection_id	string	The unique connection id to a mongo connection configuration

MssqlDestinationConnectionOptions

Name	Type	Description
truncate_table	MssqlTruncateTableConfig
init_table_schema	bool	Currently not supported and a placeholder for future implementation
on_conflict	MssqlOnConflictConfig	Currently not supported and a placeholder for future implementation
skip_foreign_key_violations	bool	Insert all valid records, skipping any that violate foreign key constraints.
batch	BatchConfig	Configure batching options to handle how much data is sent to your database at once.
max_in_flight	optional uint32	Determines the maximum number of parallel batched inserts.

MssqlOnConflictConfig

Name	Type	Description
do_nothing	bool

MssqlSourceConnectionOptions

Name	Type	Description
halt_on_new_column_addition	bool
schemas	repeated MssqlSourceSchemaOption
connection_id	string
subset_by_foreign_key_constraints	bool

MssqlSourceSchemaOption

Name	Type	Description
schema	string
tables	repeated MssqlSourceTableOption

MssqlSourceSchemaSubset

Name	Type	Description
mssql_schemas	repeated MssqlSourceSchemaOption

MssqlSourceTableOption

Name	Type	Description
table	string
where_clause	optional string

MssqlTruncateTableConfig

Name	Type	Description
truncate_before_insert	bool

MysqlDestinationConnectionOptions

Name	Type	Description
truncate_table	MysqlTruncateTableConfig
init_table_schema	bool
on_conflict	MysqlOnConflictConfig
skip_foreign_key_violations	bool	Insert all valid records, skipping any that violate foreign key constraints.
batch	BatchConfig	Configure batching options to handle how much data is sent to your database at once.
max_in_flight	optional uint32	Determines the maximum number of parallel batched inserts.

MysqlOnConflictConfig

Name	Type	Description
do_nothing	bool

MysqlSourceConnectionOptions

Name	Type	Description
halt_on_new_column_addition	bool
schemas	repeated MysqlSourceSchemaOption
connection_id	string
subset_by_foreign_key_constraints	bool

MysqlSourceSchemaOption

Name	Type	Description
schema	string
tables	repeated MysqlSourceTableOption

MysqlSourceSchemaSubset

Name	Type	Description
mysql_schemas	repeated MysqlSourceSchemaOption

MysqlSourceTableOption

Name	Type	Description
table	string
where_clause	optional string

MysqlTruncateTableConfig

Name	Type	Description
truncate_before_insert	bool

NewJobHook

Name	Type	Description
name	string	Name of the hook for display/reference.
description	string	Description of what this hook does.
config	JobHookConfig	Hook-type specific configuration.
enabled	bool	Whether or not the hook is enabled.
priority	uint32	The priority of the hook (0-100). This determines the execution order. Lower values are higher priority (priority=0 is the highest). Tie Breaking is determined by the following: (priority, created_at, id) in ascending order.

PauseJobRequest

Name	Type	Description
id	string
pause	bool
note	optional string

PauseJobResponse

Name	Type	Description
job	Job

PendingActivity

Name	Type	Description
status	ActivityStatus
activity_name	string
last_failure	optional ActivityFailure

PostgresDestinationConnectionOptions

Name	Type	Description
truncate_table	PostgresTruncateTableConfig
init_table_schema	bool
on_conflict	PostgresOnConflictConfig
skip_foreign_key_violations	bool	Insert all valid records, skipping any that violate foreign key constraints.
batch	BatchConfig	Configure batching options to handle how much data is sent to your database at once.
max_in_flight	optional uint32	Determines the maximum number of parallel batched inserts.

PostgresOnConflictConfig

Name	Type	Description
do_nothing	bool

PostgresSourceConnectionOptions

Name	Type	Description
halt_on_new_column_addition	optional bool	@deprecated - Use new_column_addition_strategy instead
schemas	repeated PostgresSourceSchemaOption
connection_id	string
subset_by_foreign_key_constraints	bool
new_column_addition_strategy	PostgresSourceConnectionOptions.NewColumnAdditionStrategy	Provide a strategy of what to do in the event Neosync encounters an unmapped column for the job's mapped tables.

PostgresSourceConnectionOptions.NewColumnAdditionStrategy

Name	Type	Description
halt_job	PostgresSourceConnectionOptions.NewColumnAdditionStrategy.HaltJob	halt job if a new column is detected. This is equiavlent to the deprecated halt_on_new_column_addition
auto_map	PostgresSourceConnectionOptions.NewColumnAdditionStrategy.AutoMap	automatically handle unmapped columns. It handles this by using the DBs default/nullable values. If this doesn't exist, will fall back to configuring generators for supported datatypes. If none of the criteria above can be met, the job run will fail to prevent leaking of PII.

PostgresSourceConnectionOptions.NewColumnAdditionStrategy.AutoMap

Configuration for the AutoMap strategy

Name	Type	Description

PostgresSourceConnectionOptions.NewColumnAdditionStrategy.HaltJob

Configuration for the HaltJob strategy

Name	Type	Description

PostgresSourceSchemaOption

Name	Type	Description
schema	string
tables	repeated PostgresSourceTableOption

PostgresSourceSchemaSubset

Name	Type	Description
postgres_schemas	repeated PostgresSourceSchemaOption

PostgresSourceTableOption

Name	Type	Description
table	string
where_clause	optional string

PostgresTruncateTableConfig

Name	Type	Description
truncate_before_insert	bool
cascade	bool

RetryPolicy

Defines the retry policy for an activity

Name	Type	Description
maximum_attempts	optional int32	Maximum number of attempts. When exceeded the retries stop even if not expired yet. If not set or set to 0, it means unlimited, and rely on activity ScheduleToCloseTimeout to stop.

RunContextKey

Name	Type	Description
job_run_id	string	The Neosync Run ID
external_id	string	An opaque identifier that will be used to store specific items
account_id	string	The Neosync Account ID

SetJobHookEnabledRequest

Name	Type	Description
id	string	The unique identifier of the hook
enabled	bool	Whether or not the hook is enabled.

SetJobHookEnabledResponse

Name	Type	Description
hook	JobHook	The updated job hook

SetJobSourceSqlConnectionSubsetsRequest

Name	Type	Description
id	string	The unique identifier of the job to update subsets for
schemas	JobSourceSqlSubetSchemas	The subset configuration
subset_by_foreign_key_constraints	bool	Whether or not to have subsets follow foreign key constraints (for connections that support it)

SetJobSourceSqlConnectionSubsetsResponse

Name	Type	Description
job	Job

SetJobSyncOptionsRequest

Name	Type	Description
id	string	The unique identifier of the job
sync_options	ActivityOptions	The sync options object. The entire object must be provided and will fully overwrite the previous result

SetJobSyncOptionsResponse

Name	Type	Description
job	Job

SetJobWorkflowOptionsRequest

Name	Type	Description
id	string	The unique identifier of the job
worfklow_options	WorkflowOptions	The workflow options object. The entire object must be provided and will fully overwrite the previous result

SetJobWorkflowOptionsResponse

Name	Type	Description
job	Job

SetRunContextRequest

Name	Type	Description
id	RunContextKey
value	bytes	An opaque value that is to be determined by the key

SetRunContextResponse

Name	Type	Description

SetRunContextsRequest

Name	Type	Description
id	RunContextKey
value	bytes	An opaque value that is to be determined by the key

SetRunContextsResponse

Name	Type	Description

TerminateJobRunRequest

Name	Type	Description
job_run_id	string
account_id	string

TerminateJobRunResponse

Name	Type	Description

UpdateJobDestinationConnectionRequest

Name	Type	Description
job_id	string
connection_id	string
options	JobDestinationOptions
destination_id	string

UpdateJobDestinationConnectionResponse

Name	Type	Description
job	Job

UpdateJobHookRequest

Name	Type	Description
id	string	The unique identifier of the hook
name	string	Name of the hook for display/reference.
description	string	Description of what this hook does.
config	JobHookConfig	Hook-type specific configuration.
enabled	bool	Whether or not the hook is enabled.
priority	uint32	The priority of the hook (0-100). This determines the execution order. Lower values are higher priority (priority=0 is the highest). Tie Breaking is determined by the following: (priority, created_at, id) in ascending order.

UpdateJobHookResponse

Name	Type	Description
hook	JobHook	The updated job hook

UpdateJobScheduleRequest

Name	Type	Description
id	string
cron_schedule	optional string

UpdateJobScheduleResponse

Name	Type	Description
job	Job

UpdateJobSourceConnectionRequest

Name	Type	Description
id	string
source	JobSource
mappings	repeated JobMapping
virtual_foreign_keys	repeated VirtualForeignConstraint

UpdateJobSourceConnectionResponse

Name	Type	Description
job	Job

ValidateJobMappingsRequest

Name	Type	Description
account_id	string	The unique account identifier that this job will be associated with
mappings	repeated JobMapping
connection_id	string
virtual_foreign_keys	repeated VirtualForeignConstraint

ValidateJobMappingsResponse

Name	Type	Description
column_errors	repeated ColumnError
database_errors	DatabaseError

VirtualForeignConstraint

Name	Type	Description
schema	string
table	string
columns	repeated string
foreign_key	VirtualForeignKey

VirtualForeignKey

Name	Type	Description
schema	string
table	string
columns	repeated string

WorkflowOptions

Config that contains various timeouts that are configured in the underlying temporal workflow More options will come in the future as needed

Name	Type	Description
run_timeout	optional int64	The timeout for a single workflow run. Measured in seconds

---

Enums

ActivityStatus

Name	Number	Description
ACTIVITY_STATUS_UNSPECIFIED	0
ACTIVITY_STATUS_SCHEDULED	1
ACTIVITY_STATUS_STARTED	2
ACTIVITY_STATUS_CANCELED	3
ACTIVITY_STATUS_FAILED	4

AwsS3DestinationConnectionOptions.StorageClass

Name	Number	Description
STORAGE_CLASS_UNSPECIFIED	0
STORAGE_CLASS_STANDARD	1
STORAGE_CLASS_REDUCED_REDUNDANCY	2
STORAGE_CLASS_GLACIER	3
STORAGE_CLASS_STANDARD_IA	4
STORAGE_CLASS_ONEZONE_IA	5
STORAGE_CLASS_INTELLIGENT_TIERING	6
STORAGE_CLASS_DEEP_ARCHIVE	7

GetActiveJobHooksByTimingRequest.Timing

Name	Number	Description
TIMING_UNSPECIFIED	0	If unspecified, returns all active job hooks
TIMING_PRESYNC	1	Only returns presync hooks
TIMING_POSTSYNC	2	Only returns postsync hooks

JobRunStatus

An enumeration of job run statuses.

Name	Number	Description
JOB_RUN_STATUS_UNSPECIFIED	0	if the job run status is unknown
JOB_RUN_STATUS_PENDING	1	the run is pending and has not started yet
JOB_RUN_STATUS_RUNNING	2	the run is currently in progress
JOB_RUN_STATUS_COMPLETE	3	the run has successfully completed
JOB_RUN_STATUS_ERROR	4	the run ended with an error
JOB_RUN_STATUS_CANCELED	5	the run was cancelled
JOB_RUN_STATUS_TERMINATED	6	the run was terminated
JOB_RUN_STATUS_FAILED	7	the run ended in failure
JOB_RUN_STATUS_TIMED_OUT	8	the run was ended pre-maturely due to timeout

JobStatus

Name	Number	Description
JOB_STATUS_UNSPECIFIED	0
JOB_STATUS_ENABLED	1
JOB_STATUS_PAUSED	3
JOB_STATUS_DISABLED	4

LogLevel

Name	Number	Description
LOG_LEVEL_UNSPECIFIED	0
LOG_LEVEL_DEBUG	1
LOG_LEVEL_INFO	2
LOG_LEVEL_WARN	3
LOG_LEVEL_ERROR	4

LogWindow

Name	Number	Description
LOG_WINDOW_NO_TIME_UNSPECIFIED	0
LOG_WINDOW_FIFTEEN_MIN	1
LOG_WINDOW_ONE_HOUR	2
LOG_WINDOW_ONE_DAY	3

---

Services

JobService

GetJobs

Method	GetJobs
Request	GetJobsRequest
Response	GetJobsResponse
Description

GetJob

Method	GetJob
Request	GetJobRequest
Response	GetJobResponse
Description

CreateJob

Method	CreateJob
Request	CreateJobRequest
Response	CreateJobResponse
Description

DeleteJob

Method	DeleteJob
Request	DeleteJobRequest
Response	DeleteJobResponse
Description

IsJobNameAvailable

Method	IsJobNameAvailable
Request	IsJobNameAvailableRequest
Response	IsJobNameAvailableResponse
Description

UpdateJobSchedule

Method	UpdateJobSchedule
Request	UpdateJobScheduleRequest
Response	UpdateJobScheduleResponse
Description

UpdateJobSourceConnection

Method	UpdateJobSourceConnection
Request	UpdateJobSourceConnectionRequest
Response	UpdateJobSourceConnectionResponse
Description

SetJobSourceSqlConnectionSubsets

Method	SetJobSourceSqlConnectionSubsets
Request	SetJobSourceSqlConnectionSubsetsRequest
Response	SetJobSourceSqlConnectionSubsetsResponse
Description

UpdateJobDestinationConnection

Method	UpdateJobDestinationConnection
Request	UpdateJobDestinationConnectionRequest
Response	UpdateJobDestinationConnectionResponse
Description

DeleteJobDestinationConnection

Method	DeleteJobDestinationConnection
Request	DeleteJobDestinationConnectionRequest
Response	DeleteJobDestinationConnectionResponse
Description

CreateJobDestinationConnections

Method	CreateJobDestinationConnections
Request	CreateJobDestinationConnectionsRequest
Response	CreateJobDestinationConnectionsResponse
Description

PauseJob

Method	PauseJob
Request	PauseJobRequest
Response	PauseJobResponse
Description

GetJobRecentRuns

Method	GetJobRecentRuns
Request	GetJobRecentRunsRequest
Response	GetJobRecentRunsResponse
Description	Returns a list of recently invoked job runs baseds on the Temporal cron scheduler. This will return a list of job runs that include archived runs

GetJobNextRuns

Method	GetJobNextRuns
Request	GetJobNextRunsRequest
Response	GetJobNextRunsResponse
Description	Returns a list of runs that are scheduled for execution based on the Temporal cron scheduler.

GetJobStatus

Method	GetJobStatus
Request	GetJobStatusRequest
Response	GetJobStatusResponse
Description

GetJobStatuses

Method	GetJobStatuses
Request	GetJobStatusesRequest
Response	GetJobStatusesResponse
Description

GetJobRuns

Method	GetJobRuns
Request	GetJobRunsRequest
Response	GetJobRunsResponse
Description	Returns a list of job runs by either account or job

GetJobRunEvents

Method	GetJobRunEvents
Request	GetJobRunEventsRequest
Response	GetJobRunEventsResponse
Description

GetJobRun

Method	GetJobRun
Request	GetJobRunRequest
Response	GetJobRunResponse
Description	Returns a specific job run, along with any of its pending activities

DeleteJobRun

Method	DeleteJobRun
Request	DeleteJobRunRequest
Response	DeleteJobRunResponse
Description

CreateJobRun

Method	CreateJobRun
Request	CreateJobRunRequest
Response	CreateJobRunResponse
Description

CancelJobRun

Method	CancelJobRun
Request	CancelJobRunRequest
Response	CancelJobRunResponse
Description

TerminateJobRun

Method	TerminateJobRun
Request	TerminateJobRunRequest
Response	TerminateJobRunResponse
Description

GetJobRunLogsStream

Method	GetJobRunLogsStream
Request	GetJobRunLogsStreamRequest
Response	GetJobRunLogsStreamResponse stream
Description	Returns a stream of logs from the worker nodes that pertain to a specific job run

SetJobWorkflowOptions

Method	SetJobWorkflowOptions
Request	SetJobWorkflowOptionsRequest
Response	SetJobWorkflowOptionsResponse
Description	Set any job workflow options. Must provide entire object as is it will fully override the previous configuration

SetJobSyncOptions

Method	SetJobSyncOptions
Request	SetJobSyncOptionsRequest
Response	SetJobSyncOptionsResponse
Description	Set the job sync options. Must provide entire object as it will fully override the previous configuration

ValidateJobMappings

Method	ValidateJobMappings
Request	ValidateJobMappingsRequest
Response	ValidateJobMappingsResponse
Description	validates that the jobmapping configured can run with table constraints

GetRunContext

Method	GetRunContext
Request	GetRunContextRequest
Response	GetRunContextResponse
Description	Gets a run context to be used by a workflow run

SetRunContext

Method	SetRunContext
Request	SetRunContextRequest
Response	SetRunContextResponse
Description	Sets a run context to be used by a workflow run

SetRunContexts

Method	SetRunContexts
Request	SetRunContextsRequest stream
Response	SetRunContextsResponse
Description	Sets a stream of run contexts to be used by a workflow run

GetJobHooks

Method	GetJobHooks
Request	GetJobHooksRequest
Response	GetJobHooksResponse
Description	Retrieves all job hooks

GetJobHook

Method	GetJobHook
Request	GetJobHookRequest
Response	GetJobHookResponse
Description	Retrieves a specific job hook

CreateJobHook

Method	CreateJobHook
Request	CreateJobHookRequest
Response	CreateJobHookResponse
Description	Creates a new job hook

DeleteJobHook

Method	DeleteJobHook
Request	DeleteJobHookRequest
Response	DeleteJobHookResponse
Description	Removes a job hook

IsJobHookNameAvailable

Method	IsJobHookNameAvailable
Request	IsJobHookNameAvailableRequest
Response	IsJobHookNameAvailableResponse
Description	Check if a specific job hook name is available

UpdateJobHook

Method	UpdateJobHook
Request	UpdateJobHookRequest
Response	UpdateJobHookResponse
Description	Updates a job hook

SetJobHookEnabled

Method	SetJobHookEnabled
Request	SetJobHookEnabledRequest
Response	SetJobHookEnabledResponse
Description	Enables or disables a job hook

GetActiveJobHooksByTiming

Method	GetActiveJobHooksByTiming
Request	GetActiveJobHooksByTimingRequest
Response	GetActiveJobHooksByTimingResponse
Description	Returns job hooks that are enabled by a specific timing. They will be sorted by priority, created_at, and id ascending.

---