I've written a simple AWS step functions workflow with a single step:
from stepfunctions.inputs import ExecutionInput
from stepfunctions.steps import Chain, TuningStep
from stepfunctions.workflow import Workflow
import train_utils
def main():
workflow_execution_role = 'arn:aws:iam::MY ARN'
execution_input = ExecutionInput(schema={
'app_id': str
})
estimator = train_utils.get_estimator()
tuner = train_utils.get_tuner(estimator)
tuning_step = TuningStep(state_id="HP Tuning", tuner=tuner, data={
'train': f's3://my-bucket/{execution_input["app_id"]}/data/'},
wait_for_completion=True,
job_name='HP-Tuning')
workflow_definition = Chain([
tuning_step
])
workflow = Workflow(
name='HP-Tuning',
definition=workflow_definition,
role=workflow_execution_role,
execution_input=execution_input
)
workflow.create()
if __name__ == '__main__':
main()
My goal is to have the train input pulled from the execution JSON provided at runtime. When I execute the workflow (from the step functions console), providing the JSON {"app_id": "My App ID"}
the tuning step does not get the right data, instead it gets a to_string representation of the stepfunctions.inputs.placeholders.ExecutionInput
. Furthermore when looking at the generated ASL I can see that the execution input was rendered as a string:
...
"DataSource": {
"S3DataSource": {
"S3DataType": "S3Prefix",
"S3Uri": "s3://my-bucket/<stepfunctions.inputs.placeholders.ExecutionInput object at 0x12261f7d0>/data/",
"S3DataDistributionType": "FullyReplicated"
}
},
...
What am I doing wrong?
Update: As mentioned by @yoodan the SDK is probably behind, so I'll have to edit the definition before calling create. I can see there is a way to review the definition before calling create, but can I modify the graph definition? how?
from AWS - Step functions, use execution input within a TuningStep
No comments:
Post a Comment