topic Re: How to check neo4j size in ThingWorx Developers

How to check neo4j size

preddy-2 — Thu, 31 Mar 2016 13:30:41 GMT

Hello all,

I created a value stream and associated it to a Thing property. And that, thing property is updated on timer event at every 5 sec.

value stream is created with default persistence provider, so I understand that data get stored in neo.

I am interested to notice the size of neo and what rate its size grows.

Can anyone please let me know, where can I check it?

With B/R,

Praveen.B

Re: How to check neo4j size

vranganathan — Mon, 25 Jul 2016 23:40:16 GMT

Check /ThingworxStorage/database/data folder. You will see bunch of database files like neostore.*.db.*

Re: How to check neo4j size

mhollenbach — Tue, 26 Jul 2016 14:53:01 GMT

To expand upon Varathan's answer, a neo database records all transactional data in logical logs in the /ThingworxStorage/database/data directory. This is done as a way to create a backup of the database in the event that ThingWorx does not shut down cleanly. ThingWorx has a default retention period of 7 days before the logs are wiped, but this can be modified in a platform-settings.json file that is described in this article, which links to the Help Center for further details.

Meghan

Re: How to check neo4j size

vranganathan — Tue, 26 Jul 2016 15:10:10 GMT

We did reduce the retention to one day last week to save space. The problem we are facing now is neostore.relationshipstore.db grown exponentially after purging the streams and value streams data. In that process it created tons of nioneo_logical.log.v# files and we cleaned them up. The neo.*.db and its one day backup takes up all the memory in the system. We are not sure why the file size grows when we purge the data and not cleaning up after the restart of tomcat or backups. How to compact the size of those files.

I would also appreciate if you could provide some sizing guidelines.

Re: How to check neo4j size

Aanjan — Tue, 26 Jul 2016 15:16:08 GMT

Varathan, if you're on a test server, can you try restarting Tomcat once and see how the behavior is? Does it start filling up as soon as the server is up (and the streams are getting written) or is it a gradual process?

Re: How to check neo4j size

vranganathan — Tue, 26 Jul 2016 15:32:28 GMT

This is the demo instance at CVC. Currently we are 100% on the machine and we are not able to do much. I cleaned up all unwanted file to complete the backup yesterday. I was hoping backup will compact the database. I did restart the tomcat several times. I also turned off Streaming into the Thingworx now. We noticed the file size is growing and taking up the space yesterday morning. We thought it is because of value streams constant data flow. I purged the Stream and Value Stream data yesterday. I validated to see the purging is complete and I don't see the older values anymore.

During the above process neostore.*.db file started growing exponentially and understandably created large number of nineo_logical.log.v# file (over 1200 files). I removed nineo_logical.log.v# on regular basis to free up the memory. But neostore.*.db is 70% more than what it was yesterday by this time. After taking backup yesterday night, the /ThingworxStorageBackup also increased by similar amount.

After completing the purge, I turned the streaming for a while, the growth is not that significant. I am positive purging caused this increased and not sure how to compact the files.

Re: How to check neo4j size

Aanjan — Tue, 26 Jul 2016 17:35:01 GMT

Couple follow up questions then - what are your Java options? Is the system memory where you have ThingWorx running atleast 8 gigs? Do you have a scheduled backup that runs everyday? And what is at a 100%? Is it the disk write or your processes?

Re: How to check neo4j size

vranganathan — Tue, 26 Jul 2016 17:49:35 GMT

Here are the options used to start tomcat

/usr/lib/jvm/jre/bin/java -Djava.awt.headless=true -Djava.net.preferIPv4Stack=true -Dserver -Dd64 -Xms512m -Xmx2g -XX:+UseNUMA -XX:+UseConcMarkSweepGC -Dfile.encoding=UTF-8 -Djava.security.egd=file:/dev/urandom -classpath /usr/share/tomcat/bin/bootstrap.jar:/usr/share/tomcat/bin/tomcat-juli.jar:/usr/share/java/commons-daemon.jar -Dcatalina.base=/usr/share/tomcat -Dcatalina.home=/usr/share/tomcat -Djava.endorsed.dirs= -Djava.io.tmpdir=/var/cache/tomcat/temp -Djava.util.logging.config.file=/usr/share/tomcat/conf/logging.properties -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager org.apache.catalina.startup.Bootstrap start

Yes. Backup runs every day. We have retention policy of 1 day.

100% indicates the disk is full. we have about 25GB for Thingworx. ThnigworxStorage and ThingworxStorageBackup took it all.There is nothing left to clean up either. Our Hard disk usage increased around 70% just yesterday. The only extra thing we did from normal operation was purging of Value Stream data. During Purging we saw the files growing significantly. That is the problem we are trying to solve.

Re: How to check neo4j size

mhollenbach — Tue, 26 Jul 2016 18:46:17 GMT

Varathan,

The recommend server sizing is 100GB dedicated hard disk space, a 16GB system, and a 4 core processor. This is of course geared towards a production server, and a demo server can certainly have less dedicated hard disk space and ram. When you say your "Hard disk usage increased around 70%", what was the initial value?

Do you have any idea as to how many records were in the ValueStream that were being purged?

Meghan

Re: How to check neo4j size

Aanjan — Tue, 26 Jul 2016 19:18:04 GMT

Adding to Meghan's response, 2 gigs set for the Xmx setting is pretty low. What is your available system RAM? What is in your BackupStorage? Is it just backups of your current ThingWorx entities? How many are there, and how big are they?

Re: How to check neo4j size

vranganathan — Tue, 26 Jul 2016 19:40:00 GMT

This is the virtual Machine running on Edge Line server. We may able to bump up the RAM. Currently it is 4GB I believe. This is just demo system and we did not experience any issues in terms of user experience. There is nothing else running on the machine. I see another 2GB is not used at all as per free command. However I am not sure how it is related db file size growth.

Here is the disk usage

11370180 ./ThingworxStorage

0 ./ThingworxPlatform

11277868 ./ThingworxBackupStorage

It is just backup of data folder. Nothing else is backed up. That too one day retention.

Re: How to check neo4j size

Aanjan — Tue, 26 Jul 2016 20:23:08 GMT

This would directly correlate to your specs; as threads gets filled up and the memory gets taxed, neo will not be able to keep up and start writing more and more logs for it to complete at a future time. This will eventually lead to Tomcat crashing or even your system becoming unresponsive. As Meghan noted, 16 gigs is necessary, and with the current 4 gigs you have, this is not enough.

Re: How to check neo4j size

vranganathan — Tue, 26 Jul 2016 20:36:34 GMT

I did realize the limitation of Neo4j. I am submitted a request to build new VM with Postgres DB, 100GB HDD and 16GB RAM. However I need to figure out something to get by until we receive it. We have customer demos scheduled on daily basis. That is where I am trying to figure out short term solution.

Re: How to check neo4j size

Aanjan — Tue, 26 Jul 2016 21:28:35 GMT

One thing to start off with is your backups - since you have backups taking space as well, you can temporarily disable them (or reduce the frequency) or change the backup type to incremental backup.

Re: How to check neo4j size

vranganathan — Tue, 26 Jul 2016 21:46:18 GMT

How do I disable the backup. Disable the timer? I am also copying the backup folder to my local machine now. I am planning to delete the content of backup folder after that. We are not updating any entities. We don't care about data coming in at this point. I am thinking if something goes wrong, I should able to restore from my local machine backup. Do you see any problem?

Once we get the new VM, we can move to Postgres.

Thanks.

Re: How to check neo4j size

Aanjan — Wed, 27 Jul 2016 15:32:24 GMT

That should work. Disabling the Scheduler works as well; I've linked 2 articles in my previous post with which you can extend the backup frequency if needed. The 2nd one would change the backup type to incremental, which means it wouldn't do a full backup every time; it will check for changes and save accordingly.