I am using a dataflow function to transform pubsub messages in the form of json written as a string to submit into a bigquery table with the correct schemas in place.
I use the following UDF function which I have tested to work correctly in the google cloud shell.
function networkprocess(inJson) {
data = JSON.parse(inJson);
data ["Network_Details"] = JSON.parse(data["Network_Details"]);
index = data["Network_Details"]["IpAddress"].indexOf(data["source_internal_ip"]);
if (index == -1) {
index = 0;
}
for (var key in data["Network_Details"]){
data[key] = (data["Network_Details"][key][index]);
}
delete data ["Network_Details"];
return JSON.stringify(data);
}
However this dataflow job keeps outputting failed records to a failed record table with the following error:
java.lang.NoSuchMethodException: No such function networkprocess at org.openjdk.nashorn.api.scripting.ScriptObjectMirror.callMember(ScriptObjectMirror.java:192) at .......
............ a lot of "at" pointers....
I have tried creating new UDFs from scracth as well as using the standard function google provides when clicking create UDF during the job set up (in GUI) which is just:
function process(inJson) {
return JSON.stringify(data);
}
and it they all return the same error that it cannot find the function.
I am deploying this via GUI or terraform but its the same result.
from Google Cloud Dataflow error NoSuchMethodException: No such function
No comments:
Post a Comment