Checkpoint pipeline: How to create and configure it?

hieuiph 2017-03-23 11:12:59 UTC #1

I create the Cumulative aggregation and I see in the guide that:

"To use the cumulative aggregation it is mandatory to configure the check point in pipeline from pipeline definition."

How can we configure the check point ?

anupam.jain 2017-03-23 11:59:19 UTC #2

When you hit save button of pipeline, "Pipeline Definition" pop up is invoked. Expand the +More button and set "Enable Check Point" true and set the "Check Point Directory" path as below screen shot

hieuiph 2017-03-23 13:18:49 UTC #3

Hi,

I followed your suggestion but I can't change the filed "Enable Check Point" from "FALSE" to "TRUE", it's deactivated.

thanks

anupam.jain 2017-03-23 13:28:01 UTC #4

Your pipeline is in running mode, thats why options are disabled, kindly stop, edit and save the pipeline and run.

hieuiph 2017-03-23 13:56:59 UTC #5

it works. I will try and let you know if I have a problem.

Thanks

hieuiph 2017-03-23 15:57:52 UTC #6

I created cumulative aggregation as below:

the filter's configuration:

and the cumulative's configuration is:

When inspect pipeline, I get the error:

Please advice.

thanks

hieuiph 2017-03-24 07:08:41 UTC #7

Hello,

what's is '/path_to_dir/ in my case ?

thanks

Mohit_Bhindwal 2017-03-24 07:16:02 UTC #8

Hi hieuiph

It's a path used by pipeline to run, it can be any random path like /hieuiph1 please update the path in case of re run.

hieuiph 2017-03-24 07:20:28 UTC #9

Hi,

thanks. you mean that we must update the path in 'pipeline definition' when we would like to re run inspect pipeline ?

thanks

Mohit_Bhindwal 2017-03-24 07:40:38 UTC #10

In case of inspect it will ask for its own check point directory if inspect require it.
you don't need to update check point directory in pipeline definition.

hieuiph 2017-03-24 07:57:12 UTC #11

so what case we have to set different "Check Point Directory" path with each run. ?

thank you

Mohit_Bhindwal 2017-03-24 09:00:51 UTC #12

In case you are running a pipeline:
If you set the checkpoint directory, you need to update checkpoint directory for every consecutive run.

In case you are inspecting a pipeline:
The checkpoint option will automatically appear on 'Inspect Definition' if you are using cumulative aggregation processor or TransformByKey with updateStateByKey function.Please update it for every consecutive run.