Monday 8 May 2023

Google Cloud Dataflow error NoSuchMethodException: No such function

I am using a dataflow function to transform pubsub messages in the form of json written as a string to submit into a bigquery table with the correct schemas in place.

I use the following UDF function which I have tested to work correctly in the google cloud shell.

function networkprocess(inJson) {
  data = JSON.parse(inJson);

  data ["Network_Details"] = JSON.parse(data["Network_Details"]);
  index = data["Network_Details"]["IpAddress"].indexOf(data["source_internal_ip"]);
  if (index == -1) {
      index = 0;
    }
  for (var key in data["Network_Details"]){
      data[key] = (data["Network_Details"][key][index]);
    }
  

  delete  data ["Network_Details"];
 return JSON.stringify(data);
}

However this dataflow job keeps outputting failed records to a failed record table with the following error:

java.lang.NoSuchMethodException: No such function networkprocess at org.openjdk.nashorn.api.scripting.ScriptObjectMirror.callMember(ScriptObjectMirror.java:192) at .......

............ a lot of "at" pointers....

I have tried creating new UDFs from scracth as well as using the standard function google provides when clicking create UDF during the job set up (in GUI) which is just:

function process(inJson) {

 return JSON.stringify(data);
}

and it they all return the same error that it cannot find the function.

I am deploying this via GUI or terraform but its the same result.



from Google Cloud Dataflow error NoSuchMethodException: No such function

No comments:

Post a Comment