Mynodbcsv: Lightweight Zero-Config Database Solution for Handling Very Large CSV Files.
Détails
Etat: Public
Version: de l'auteur⸱e
ID Serval
serval:BIB_B8B49FA6CD96
Type
Article: article d'un périodique ou d'un magazine.
Collection
Publications
Institution
Titre
Mynodbcsv: Lightweight Zero-Config Database Solution for Handling Very Large CSV Files.
Périodique
Plos One
ISSN
1932-6203 (Electronic)
ISSN-L
1932-6203
Statut éditorial
Publié
Date de publication
2014
Peer-reviewed
Oui
Volume
9
Numéro
7
Pages
e103319
Langue
anglais
Notes
Publication types: Journal Article Publication Status: epublish
Résumé
Volumes of data used in science and industry are growing rapidly. When researchers face the challenge of analyzing them, their format is often the first obstacle. Lack of standardized ways of exploring different data layouts requires an effort each time to solve the problem from scratch. Possibility to access data in a rich, uniform manner, e.g. using Structured Query Language (SQL) would offer expressiveness and user-friendliness. Comma-separated values (CSV) are one of the most common data storage formats. Despite its simplicity, with growing file size handling it becomes non-trivial. Importing CSVs into existing databases is time-consuming and troublesome, or even impossible if its horizontal dimension reaches thousands of columns. Most databases are optimized for handling large number of rows rather than columns, therefore, performance for datasets with non-typical layouts is often unacceptable. Other challenges include schema creation, updates and repeated data imports. To address the above-mentioned problems, I present a system for accessing very large CSV-based datasets by means of SQL. It's characterized by: "no copy" approach - data stay mostly in the CSV files; "zero configuration" - no need to specify database schema; written in C++, with boost [1], SQLite [2] and Qt [3], doesn't require installation and has very small size; query rewriting, dynamic creation of indices for appropriate columns and static data retrieval directly from CSV files ensure efficient plan execution; effortless support for millions of columns; due to per-value typing, using mixed text/numbers data is easy; very simple network protocol provides efficient interface for MATLAB and reduces implementation time for other languages. The software is available as freeware along with educational videos on its website [4]. It doesn't need any prerequisites to run, as all of the libraries are included in the distribution package. I test it against existing database solutions using a battery of benchmarks and discuss the results.
Pubmed
Web of science
Open Access
Oui
Création de la notice
05/09/2014 18:13
Dernière modification de la notice
20/08/2019 16:26