No Such Function Error for $lookHBase

kalyanischakravarthi 2017-02-24 05:31:47 UTC #1

I tried all possibilities Am getting below error for enrich processor while trying to pull geopoint from hbase table.

Logged in as: dr.who
Logs for container_e05_1487064224738_0135_01_000002
ResourceManager
RM Home
NodeManager
Tools
Showing 4096 bytes. Click here for full log
k.rdd.PairRDDFunctions$$anonfun$saveAsNewAPIHadoopDataset$1$$anonfun$12$$anonfun$apply$4.apply(PairRDDFunctions.scala:1111)
at org.apache.spark.rdd.PairRDDFunctions$$anonfun$saveAsNewAPIHadoopDataset$1$$anonfun$12$$anonfun$apply$4.apply(PairRDDFunctions.scala:1111)
at org.apache.spark.util.Utils$.tryWithSafeFinallyAndFailureCallbacks(Utils.scala:1277)
at org.apache.spark.rdd.PairRDDFunctions$$anonfun$saveAsNewAPIHadoopDataset$1$$anonfun$12.apply(PairRDDFunctions.scala:1119)
at org.apache.spark.rdd.PairRDDFunctions$$anonfun$saveAsNewAPIHadoopDataset$1$$anonfun$12.apply(PairRDDFunctions.scala:1091)
at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:66)
at org.apache.spark.scheduler.Task.run(Task.scala:89)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:227)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
2017-02-23 21:27:06 Executor task launch worker-2 [ERROR] UDFUtils - Error while evaluating mvel Expresion: [Error: no such method or function: entest3Enricher_351526lookupHBase]
[Near : {... e('ip2geo',NetworkConnectionDetails.ip,'geopoint')[0].get('geocoordinates') ....}]
^
[Line: 1, Column: 83]
[Error: no such method or function: entest3Enricher_351526lookupHBase]
[Near : {... e('ip2geo',NetworkConnectionDetails.ip,'geopoint')[0].get('geocoordinates') ....}]
^
[Line: 1, Column: 83]
at org.mvel2.PropertyAccessor.getMethod(PropertyAccessor.java:897)
at org.mvel2.PropertyAccessor.getNormal(PropertyAccessor.java:181)
at org.mvel2.PropertyAccessor.get(PropertyAccessor.java:145)
at org.mvel2.PropertyAccessor.get(PropertyAccessor.java:125)
at org.mvel2.ast.ASTNode.getReducedValue(ASTNode.java:187)
at org.mvel2.MVELInterpretedRuntime.parseAndExecuteInterpreted(MVELInterpretedRuntime.java:101)
at org.mvel2.MVELInterpretedRuntime.parse(MVELInterpretedRuntime.java:47)
at org.mvel2.MVEL.eval(MVEL.java:136)
at com.streamanalytix.udf.util.UDFUtils.evaluateExpresion(UDFUtils.java:87)
at com.streamanalytix.udf.executor.UDFExecuter.execute(UDFExecuter.java:187)
at com.streamanalytix.spark.processor.EnrichProcessor$1.execute(EnrichProcessor.java:99)
at com.streamanalytix.spark.core.functions.SaxFunction.call(SaxFunction.java:93)
at org.apache.spark.api.java.JavaPairRDD$$anonfun$toScalaFunction$1.apply(JavaPairRDD.scala:1015)
at scala.collection.Iterator$$anon$11.next(Iterator.scala:328)
at scala.collection.Iterator$$anon$11.next(Iterator.scala:328)
at scala.collection.Iterator$$anon$11.next(Iterator.scala:328)
at scala.collection.Iterator$$anon$11.next(Iterator.scala:328)
at org.apache.spark.rdd.PairRDDFunctions$$anonfun$saveAsNewAPIHadoopDataset$1$$anonfun$12$$anonfun$apply$4.apply$mcV$sp(PairRDDFunctions.scala:1112)
at org.apache.spark.rdd.PairRDDFunctions$$anonfun$saveAsNewAPIHadoopDataset$1$$anonfun$12$$anonfun$apply$4.apply(PairRDDFunctions.scala:1111)
at org.apache.spark.rdd.PairRDDFunctions$$anonfun$saveAsNewAPIHadoopDataset$1$$anonfun$12$$anonfun$apply$4.apply(PairRDDFunctions.scala:1111)
at org.apache.spark.util.Utils$.tryWithSafeFinallyAndFailureCallbacks(Utils.scala:1277)
at org.apache.spark.rdd.PairRDDFunctions$$anonfun$saveAsNewAPIHadoopDataset$1$$anonfun$12.apply(PairRDDFunctions.scala:1119)
at org.apache.spark.rdd.PairRDDFunctions$$anonfun$saveAsNewAPIHadoopDataset$1$$anonfun$12.apply(PairRDDFunctions.scala:1091)
at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:66)
at org.apache.spark.scheduler.Task.run(Task.scala:89)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:227)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)

Help is very much appreciated.

Regards
Kalyan

sameerb 2017-02-24 07:02:07 UTC #2

Hello,

The function name is lookupHBase

anupam.jain 2017-02-24 09:22:58 UTC #3

Greetings!

Could you please share the complete mvel expression you are using in enricher, screen-shot would also be helpful.

Thanks

kalyanischakravarthi 2017-02-24 16:33:11 UTC #4

Hello Sameer,

Thanks for response. I have used the same function. Indeed i didnt type the function I have selected it from dropdown in tool.

Regards
Kalyan

kalyanischakravarthi 2017-02-24 16:40:10 UTC #5

sameerb 2017-02-24 19:26:05 UTC #6

Hello Kalyan,

Can you please save your pipeline so that it will be easier for us to debug? It looks like you are using the inspect feature to test your pipelines without saving them.
The view Hbase page works fine for me when I transition to your new workspace. It looks like an issue when your workspace was migrated. In the meantime please use the ip2geo schema posted in the forum.
I will have the view Hbase & MVEL issue fixed in the meanwhile.
Once you save a pipeline - my team can take a look at the issues that you are facing and get back to you.

Thanks for your patience !
Sameer

kalyanischakravarthi 2017-02-24 20:36:05 UTC #7

Hello Sameer,

As mentioned by you i have saved my pipeline in workspace "kalyanischakravarthi". Please take a look. this time it looks like it has worked as i dont see error message in executor log and i can see sucessful connection established to Hbase.

But i cant confirm it becasue i cant veiw data using " view data" feature.

Not sure why its not working for me. I still cant see data in my new workspace.

Was really exited of this tool as we have lot of streaming use cases at Dish/Sling. Now i started loosing hope.

punit 2017-02-25 01:34:18 UTC #8

Hello Kalyan,
Thank you for writing to us and letting us know the issue you are facing.
We are currently looking into it. We see permission issue for your user with HBase. I have requested the Hadoop infrastructure team to get this resolved on priority.
Will keep you posted.

Thanks
Punit

punit 2017-02-25 19:18:07 UTC #9

Hello Kalyan,

The permission issue has been resolved. You will be able to query the table.
Thank you being patient.
Please let us know if we can help with any other thing.

Thanks

kalyanischakravarthi 2017-02-25 20:16:05 UTC #10

Hello Punit,

Looks like permission issue is not fully resolved. Am able to see data in Hbase table but when i launch pipeline it looks like it is not able to start container becasue of Invalid Token.

xception in thread "ContainerLauncher-0" java.lang.Error: org.apache.spark.SparkException: Exception while starting container container_e05_1487064224738_0163_02_000002 on host localhost
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1151)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.apache.spark.SparkException: Exception while starting container container_e05_1487064224738_0163_02_000002 on host localhost
at org.apache.spark.deploy.yarn.ExecutorRunnable.startContainer(ExecutorRunnable.scala:125)
at org.apache.spark.deploy.yarn.ExecutorRunnable.run(ExecutorRunnable.scala:68)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
... 2 more
Caused by: org.apache.hadoop.security.token.SecretManager$InvalidToken: Given NMToken for application : appattempt_1487064224738_0163_000002 is not valid for current node manager.expected : beth.us.impetus.com:45454 found : localhost:45454
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
at org.apache.hadoop.yarn.ipc.RPCUtil.instantiateException(RPCUtil.java:53)
at org.apache.hadoop.yarn.ipc.RPCUtil.unwrapAndThrowException(RPCUtil.java:104)
at org.apache.hadoop.yarn.api.impl.pb.client.ContainerManagementProtocolPBClientImpl.startContainers(ContainerManagementProtocolPBClientImpl.java:99)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:258)
at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:104)
at com.sun.proxy.$Proxy45.startContainers(Unknown Source)
at org.apache.hadoop.yarn.client.api.impl.NMClientImpl.startContainer(NMClientImpl.java:203)
at org.apache.spark.deploy.yarn.ExecutorRunnable.startContainer(ExecutorRunnable.scala:122)
... 4 more
Caused by: org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.token.SecretManager$InvalidToken): Given NMToken for application : appattempt_1487064224738_0163_000002 is not valid for current node manager.expected : beth.us.impetus.com:45454 found : localhost:45454
at org.apache.hadoop.ipc.Client.call(Client.java:1455)
at org.apache.hadoop.ipc.Client.call(Client.java:1392)
at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:229)
at com.sun.proxy.$Proxy44.startContainers(Unknown Source)
at org.apache.hadoop.yarn.api.impl.pb.client.ContainerManagementProtocolPBClientImpl.startContainers(ContainerManagementProtocolPBClientImpl.java:96)
... 13 more

punit 2017-02-26 00:53:01 UTC #11

Let me take a look and get back

anupam.jain 2017-02-26 10:40:49 UTC #12

Hi Kalyan

Permission issue is fixed.
In case of inspect pipeline, please always choose Data Source as "Local" instead of "Configured Source", along with data option as "Inline". You also need to provide the number of records on "Number of Records" text box, same number of records needs to be provided in "Data" text area. Please see attached screen-shot for your reference.

kalyanischakravarthi 2017-02-26 16:48:05 UTC #13

Hello Anupam,

Thanks for that. Still i have permission issue to view Data in HDFS. I have put emitter as HDFS and pipeline ran sucessfully but i see that i cant see output Data.

anupam.jain 2017-02-27 08:58:30 UTC #14

Hi Kalyan,

Kindly check now by running your pipeline and then click on "View Data".
For saving and viewing data on the hdfs you need to submit pipeline to the cluster.

Note: In case pipeline is running in inspect mode, StreamAnalytix will not actually save data to hdfs, as of now it ignores data persist operations for inspect mode.