Import and export pipelines

Creating a data pipeline from scratch can be time-consuming, especially when we need to perform the same merge or transformation step on a similar log. To streamline this process, we can now reuse the merge and transformation operations from previously created pipelines.

In Apromore 10.1, we can export a data pipeline (which stores the merge and transformation operations) and import the pipeline on a similar log to reuse the merge and transformation operations.

Imagine that we are creating a data pipeline. In the Merge step, we append two logs.

In the Transform step, we replace “Credit” in the Card Type column with “Postpaid” and drop rows where the dispute amount is less than 4.

But in another data pipeline, we need to perform the same merge and transformation we have just created here. Perhaps a second user wishes to create the same data pipeline using the same event logs. It could also be that extracted files get updated, and the user wishes to recreate the data pipeline on the updated file. First, we ensure we save the data pipeline.

In the Transform step, click Export to download the data pipeline for future use.

Note

We can also export the data pipeline by clicking Export in the Merge Pipeline window.

The pipeline is downloaded as a JSON file.

Now, we can create another data pipeline that utilizes the merging and transformation operation in the earlier pipeline. In the new pipeline, extract the logs to create the data pipeline.

Note

The extracted logs in the new data pipeline must have the same file names and column names as the logs from which the data pipeline was downloaded.

In the Merge step, click Import to upload the downloaded data pipeline JSON.

The same merge operation will be performed automatically.

The transformation operation will also be performed automatically.

We can perform any further actions to merge or transform the data before loading it into Apromore.