server configuration

Database

Nimbits server ships with an embedded H2 Database. You can scale this up to any database architecture compatible with Data Nucleus. In order to change the target database, edit the file: /webapp/WEB-INF/applicationContext.xml and change the bean pmf to meet your needs. Consult your database providers documentation for driver classes and requirements.

<bean id="pmf" class="org.springframework.orm.jdo.LocalPersistenceManagerFactoryBean">
        <property name="jdoProperties">
            <props>
                <prop key="javax.jdo.PersistenceManagerFactoryClass">
                    org.datanucleus.api.jdo.JDOPersistenceManagerFactory
                </prop>
                <prop key="javax.jdo.option.ConnectionURL">
                    jdbc:h2:./database/db2;MV_STORE=FALSE;MVCC=FALSE;;FILE_LOCK=NO
                </prop>
                <prop key="javax.jdo.option.ConnectionUserName">sa</prop>
                <prop key="javax.jdo.option.ConnectionPassword"></prop>
                <prop key="javax.jdo.option.ConnectionDriverName">org.h2.Driver</prop>
                <prop key="org.jpox.autoCreateSchema">true</prop>        
                <prop key="org.jpox.identifier.case">PreserveCase</prop>
                <prop key="datanucleus.autoCreateTables">true</prop>
                <prop key="datanucleus.autoCreateColumns">true</prop> 
            </props>
        </property>
    </bean>

Database Caching

You can also configure caching levels to meet your needs by configuring the above file with database level caching per this documentation: http://www.datanucleus.org/products/datanucleus/jdo/cache.html

File System

Nimbits is designed with the philosophy that while all data is important, most is never used. Therefore, we should use as little resources as possible when dealing with data until it's determined that it's important.

After a value is posted to the API, and it's checked against filters and other rules and actually make it to storage, it's written immediately to the file system as a single uncompressed file containing the json representation of the value. It's timestamp is used to store it in a directory based on the day containing that timestamp.

At this point, the system will never touch the data again unless it's to delete the folders because of an expiration date setting. When a series of data is requested, nimbits reads the values into memory and at that point, based on our algorithms, may optimize the data for efficiency on the next request, delete the files and save the optimized data in it's place.

By storing data using our time based algorithm, we're then able to read data quickly without having to perform a lookup in a database, which would take time and resources to do.

Fields in the json data are minimized for efficiency

{
     t: timestamp
     d: double value
     lt: latitude
     lg: longitude
     dx: text data
     m: meta data
}

We have considered adding a gzip process to file storage to reduce space but determined that drive space is cheaper than CPU time and it's better to keep files uncompressed rather than add to the resources needed to read it.

The location of the files is set in the server settings section of the web console.