Simple Document Store

a noSql simple JSON documents database for projects in developement.

Why

I’m often working on projects which don’t store a big amount of data during dev. I wanted to have a small store easy to use and to implement.
This is NOT for production.

What is provided or not

• no client-server architecture, database is a local file.
• manage small amount of documents, several hundreds records/documents. Whole documents are loaded in memory.
• efficient if you read much more than you write data.
• no concurrent access, in particular in writes/updates.
• no primary key (yet?).
• no indexes.
• noSql, no structure in db, you can store any javascript object, it’s up to you to add ‘type’ field if you need collection.
• search by scalar field.
• somehow human readable as file is a stack of JSON. Separator is UNICODE Record Separator (‘\u241E’).
• using unicode record separator lets you use ‘new line’ in string data field.
• you can use text editor to search and update (if you are an adventurer :-) ).
• 1 constructor and a few methods: get/search, add, save, append.

How to use it

const Store = require('simple-document-store');
const store = new Store('tests/adresses.sds');
store.get({ fieldname: 'name', value: 'Place', operator: 'startsWith' }, function (err, records) {
    if (err) {
        console.log(err);
    } else {
        // need to proceed action to load objets in memory.
        console.log('total in file: ' + store.documents.length);
        console.log('total found: ' + records.length);
    }
});

API

work in progress still testing the code. but you can give it a try and make me feedback :). Code is short and commented.

constructor

const store = new Store(path);

path
the path of the document base database.

If the file does not exist, it is created.

get

Get/search for documents.

store.get(parameters, function(err, documents));

parameters
it is an object with 3 properties

• fieldname the name of the property to evaluate
• value the value used to evaluate the propetrty
• operator (optional) the type of operation: ‘equals’, ‘startsWith’, ‘contains’. ‘equals’ by default if omitted.

callback (err, documents)
the callback returns an array of documents corresponding to the search

add

Add a document in memory but does not save it to db file. Use it to add several documents in a loop before save all in one go.

for(let i=0; i< 10;i++){
    store.add({
        id:i,
        label: 'label' + i
    });
}
store.save(funtion(err){
    console.log(err);
});

append

Add a document at end of database asynchronously. Care, using ‘append’ in a loop will probably cause a ‘file lock’ error.

store.append(object, function(err){
    console.log(err);
});

object
the object to add.

callback(err)
err is not null if something went wrong.

save

Save whole documents (which are in memory) to disk asynchronouly.

store.save(object, function(err){
    console.log(err);
});

update

It is NOT a method. you can directly update results of a search and modify documents returned, then save.

Performance

with a 480 documents database of postal adresses with geocoordonates (1 city), looking for a street took less than 1ms on a i5 8th generation Intel proc and SSD. File size is 55kb.

In comparison, the same request, with same data on a MySql database located in a datacenter took between 40 and 50ms.

I planned to do some test with a local MySQL server.

Security

No network access, the security is associated with the security of the file system the db file is located on.

TODO

• write better tests
• make backup of db file before overwrite.
• add parameters in constructor for record separator or file version rotation for instance.

Probably

• some file accesses use synchronous node js functions. Evaluate opportunities to use async ones or go to promises.
• manage primary key.
• regex in search

If i need it, this is just ideas. may be a “Less Simple Document Store”.

• define a scalable file format with pages.
• manage concurrent access.
• implements indexes with b-tree.
• Evaluate opportunities to crypt db file.
• Evaluate opportunities to compress db file or some records.