I have a small dataset consisting of around 200 measurements of different light sources. The light is measured with a broadband and an IR diode, and the resulting voltage is sampled it with 4 megasamples/second. All the measurements have a length of 2 seconds, so twice 8e6 values per measurement. At the moment, all the measurements are in my file system on Windows, structured (and labelled) by folder- and file names. Now my first question is, generally, what is the state-of-the-art way to store such data? The data is quite homogeneous, so I assume an SQL Database would fit, but then, do I store the measurement values in the DB as well, or just the paht to them? And if I store just the paths, how do I avoid the problem that they are specific to my machine? By storing the data online? Then my question is, is there a service (preferably free) which fits my needs?
I know this question is opinion-based, but maybe someone can give me a quick overview so I know what to look for and where it makes sense to start.
EDIT (Thanks Arun Aniyan) -------------------------------------------------------------
My data is stored in HDF5 files, and I'm processing it with Python 3. The ligh I'm measuring comes from welding machines, so the metadata are something like: Brand (e.g. Fronius), Process (TIG, MAG, ...), AC/DC, Amperage (10-300 Ampère), Distance to light source, date and time of measurement, info about the ambient light source(sun, LED, Halogen, ...).
At the moment, my measurements are specified by really long names (brand_process_ac/dc_amperage_distance_etc.mat). When I want to add a new category to the metadata, I have to change all the file names. Also if I want to query and filter my data (if I want for example to only consider fronius machines), I have to parse the file names. I'm not working with big data at the moment, so it works. However, I would prefer to use the right tools from the start, so my dataset is scalable and stays manageable when it grows.
