fast and efficient recording of data

[laurentperrinet]

At the moment, we are using pickle files to store neo objects, which is overkill when dealing with simple outputs from experiments

There are multiple alternatives

• stay with pickle files
• move to pandas
• elephant / mozaik / ... ?

The solution should take into account efficiency (=> benchmark) vs ease of use (=> notebook showing usage) vs generality .

[JuliaSprenger]

I checked the PyNN requirements and it looks like it should be straight forward to extend this also to the currently recommended output format nix (Nix), e.g. in these lines. This does not require the usage of nix directly as it is used only indirectly via Neo.

I think we should dig deeper into this during the next project meeting.

[laurentperrinet]

doh! not supported on my ubuntu 20.04LTS... will try to compile from source

[laurentperrinet]

I began with a typical usecase to get an ouput neo object... work in progress (after lunch!)

[JuliaSprenger]

@laurentperrinet You can use nix from neo directly if you install it via pip install neo[nixio]. No need to install the C++ implementation of nix.

[laurentperrinet]

some documentation from pyNN on writing data, using nixio in particular

[laurentperrinet]

this saved me quite a lot of time - and works very fine!

$ python3 -m pip install --user neo[nixio]
Requirement already satisfied: neo[nixio] in /usr/local/lib/python3.8/dist-packages/neo-0.8.0-py3.8.egg (0.8.0)
Requirement already satisfied: numpy>=1.7.1 in ./.local/lib/python3.8/site-packages (from neo[nixio]) (1.19.2)
Requirement already satisfied: quantities>=0.9.0 in /usr/local/lib/python3.8/dist-packages/quantities-0.12.4-py3.8.egg (from neo[nixio]) (0.12.4)
Requirement already satisfied: numpy>=1.7.1 in ./.local/lib/python3.8/site-packages (from neo[nixio]) (1.19.2)
Requirement already satisfied: quantities>=0.9.0 in /usr/local/lib/python3.8/dist-packages/quantities-0.12.4-py3.8.egg (from neo[nixio]) (0.12.4)
Collecting nixio>=1.5.0b2
  Downloading nixio-1.5.0b6-py3-none-any.whl (122 kB)
     |████████████████████████████████| 122 kB 4.8 MB/s
Requirement already satisfied: six in /usr/lib/python3/dist-packages (from nixio>=1.5.0b2->neo[nixio]) (1.14.0)
Requirement already satisfied: numpy>=1.7.1 in ./.local/lib/python3.8/site-packages (from neo[nixio]) (1.19.2)
Collecting h5py
  Downloading h5py-3.1.0-cp38-cp38-manylinux1_x86_64.whl (4.4 MB)
     |████████████████████████████████| 4.4 MB 8.5 MB/s
Requirement already satisfied: numpy>=1.7.1 in ./.local/lib/python3.8/site-packages (from neo[nixio]) (1.19.2)
Installing collected packages: h5py, nixio
Successfully installed h5py-3.1.0 nixio-1.5.0b6

[laurentperrinet]

hi @albertoarturovergani !

I re-organized the structure of the folder to separate the different sub-tasks -> e56b560

I suggest your rename dev to output and remove the prepended 3_ on the notebooks to make things simpler

[laurentperrinet]

I recommend pushing regularly your latest notebooks