topic Re: Create parquet file from Thingworx Extension in ThingWorx Developers

Create parquet file from Thingworx Extension

gfontana — Fri, 04 Nov 2022 17:35:04 GMT

Hello everyone,

I've created an extension that convert a file from csv to parquet and send it to Event Hub.

The creation of parquet is based on spark-3.3.0-bin-hadoop2 libraries installed on file system and referred by an enviroment variable.

This scenario is working well but i wondering if is possible perform the same action loading the spark-3.3.0-bin-hadoop2 as a Thingworx extension. In this case how can I referred the library in an environment variables ?

Answering question could solve the problem when Thingworx is installed in a cloud environment.

Thank you

Re: Create parquet file from Thingworx Extension

VladimirRosu — Tue, 08 Nov 2022 13:17:56 GMT

Is the code you're running with that library referencing any existing Hadoop installation? In other words, does that library needs access to an existing Apache Hadoop installation?

Re: Create parquet file from Thingworx Extension

gfontana — Wed, 09 Nov 2022 07:42:35 GMT

Hello Vladimir,

I've discovered that, in reality, if you include the right library in your project you can avoid to referred the Hadoop installation.

It's a forcing because the Hadoop installation is always searched, when you generate the parquet file, but if it's missing the writer use those jar referred locally; so the extension works well.

Thank you for your answer and have a nice day

Giorgio Fontana

Re: Create parquet file from Thingworx Extension

VladimirRosu — Wed, 09 Nov 2022 07:54:19 GMT

Very good information. So, did you manage to use that library without having any conflict with the built-in ThingWorx JAR libraries? Usually external JAR libraries also have some dependencies, which in turn can conflict with the ThingWorx libraries (but not necessarily). I looked at the library you mentioned and it had 240 JAR libraries in there as dependencies. If it works as an extension this means that all those libraries are already included in ThingWorx, which is a very lucky situation.

Re: Create parquet file from Thingworx Extension

gfontana — Wed, 09 Nov 2022 08:00:45 GMT

In my experience, developing the extension with eclipse, every time you include some external jars and run the extension is the system that notify those conflict.

So it's enough remove them from your project and the game is solved. This is happened also in this case, is a boring works but at the end it worth.

Thank you (also for the kudos)

Re: Create parquet file from Thingworx Extension

VladimirRosu — Wed, 09 Nov 2022 08:05:24 GMT

(your welcome)

Still, you're in a lucky situation, because if there is a conflict, and the library you would try to use requires a newer version of the JAR library then the one you have in ThingWorx, you could possible have a blocker (eg: if a method signature changed). Meaning that if there's significant differences you could reach a blocker.

For future users, if I would be in such a situation, I would package the extension as a JAVA Edge SDK implementation (effectively very small difference to an extension SDK, but you need to deploy it somewhere and make sure it runs )

Re: Create parquet file from Thingworx Extension

gfontana — Wed, 09 Nov 2022 08:09:12 GMT

your absolute right