Databricks + StreamAnalytix

[sobe]

Hi,
Is there a way to integrate StreamAnalytix workflow in Databricks?

Can I invoke StreamAnalytix workflow in Databricks? Can I convert StreamAnalytix workflow in python program?

There is a utility called Databricks connect that can be used to invoke Python program.

Databricks-Connect - FINALLY! — Data Thirst

Can we convert whole StreamAnalytix workflow into Python program?

Thanks
Sobe

[Mohit_Bhindwal]

Hi Sobe,

Thanks for the posting the query, Please find the responses of your questions below.

Q1. Is there a way to integrate StreamAnalytix workflow in Databricks?

StreamAnalytix has a cloud version which is available on AWS Marketplace with 15 days trial period, it enables user to configure Spark job on Databricks.
Here is the link for StreamAnalytix on Marketplace : https://aws.amazon.com/marketplace/pp/B07R2CGTPK

Q2. Can I invoke StreamAnalytix workflow in Databricks

Yes, once a StreamAnalytix job is configured on Databricks, you can run it from StreamAnalytix or Databricks.

Q3. Can I convert StreamAnalytix workflow in python program?

StreamAnalytix has a Python Connector/Processor, you can integrate your python code in it and make python code part of your StreamAnalytix workflow.

Q4. Can we convert whole StreamAnalytix workflow into Python program?

StreamAnalytix workflow can not be converted to Python program but python connector can be leveraged to execute your python code on Databricks cluster.

Regards,
Mohit Bhindwal

[sobe]

Good day Mohit. Many thanks for your prompt response.

Do you have any sample Databricks or iPython Jupyter notebook that shows how it integrates StreamAnalytix workflow?

I have gone through the videos and product desc but I did not find any actual example or relevant information how to integrate StreamAnalytix in Databricks 4.0.1 notebook.

Many thanks again for your time.

Br
Sobhan

[streamanalytix]

Hi Sobe,

To integrate with Databricks you will require to launch StreamAnalytix from AWS Marketplace(It has a 15 day free trial available), you can not do that with StreamAnalytix Lite.

AWS Marketplace URL for same: https://aws.amazon.com/marketplace/pp/B07R2CGTPK

Documentation for same : http://docs.streamanalytix.com/4.0.1/Cloud/UserGuide/

Once you launch StreamAnalytix from AWS Marketplace. You can opt one of the following ways to deploy or run jobs on Databricks:

1) Create your own data flow or pipeline in StreamAnalytix by reading from source, applying transformations and attaching sink to persist the data at a store.
Configure & run the job over databricks from within the Streamanalytix.
You don't need to access databricks Console or create any Jupyter/Databricks notebook.

OR

2) You can create a data flow in streamanalytix.
which will look like : Read from Source -> Python Processor -> Emit to a sink
2.1) Read from Source : Select a source of your choice and configure it to read the data
2.2) Python Processor : Copy your notebook code and paste it in python processor.Remove the part where you are reading data in notebook as it is coming from source stream configured in section 2.1
2.3) Emit to a sink : Connect Python processor to sin kof your choice to persist data.
Save the data flow.
Now you can configure & run the job over databricks from within the Streamanalytix.
You don't need to access databricks Console or create any Jupyter/Databricks notebook.

Let us know if this does not clarify, we can get into a call as well.