Upload of timeseries gives an error when using comma values

[FredericHirschmueller]

When uploading a timeseries that uses "," instead of "." as a decimal seperator, the timeseries is not uploaded properly. The following timeseries gives an example where the error occurs.

ninja_pv_52.1315_11.6401_corrected.csv

[paulapreuss]

From what I can see, the code to handle csv upload is handled here

gui/app/projects/helpers.py

Line 540 in f5a4e85

def parse_csv_timeseries(file_str):

The parser checks the contents of the file and based on it decides on a delimiter to set for the csv reader (, vs ;).
The csv here (for upload on a pv component) has one column and uses commas as decimal point separators. Since it only has one column, it cannot check directly for ; separators, so this check fails:

    if file_str.count(";") > 0:
        delimiter = ";"

However, I would assume that this check would actually work to parse the file correctly

    # check if the number of , is an integer time the number of line return
    # if not, the , is probably not a column separator and a decimal separator indeed
    if file_str.count(",") % (file_str.count("\n") + 1) != 0:
        delimiter = ";"

but clearly something is not quite working as intended.

@josihoppe can you try and fix this?

[Bachibouzouk]

To me the problem is that CSV means "commas separated values", therefore the commas are separators and if commas are used within the values the values should be surrounded by ", like

"1,928","2,34"
"1,064","1,01"
...

This would be easier if the file format would be .SCSV for "semi column separated values", this way one would directly know that the separator is ; and not , However this would require a big system change as most people unconsistently use CSV format with ; as separator because a certain multinational thought it would be more convenient to use ; as separators.

(Sorry for the rant, the only thing I can do is try to raise awarness in this case :D)

We should keep track of all the files which weren't properly parsed and write tests that check the file should be properly parsed, otherwise we might break a successful parsing of another oddly formatted CSV file while fixing this case.

[paulapreuss]

To me the problem is that CSV means "commas separated values", therefore the commas are separators

Agree :D

This would be easier if the file format would be .SCSV for "semi column separated values", this way one would directly know that the separator is ; and not , However this would require a big system change as most people unconsistently use CSV format with ; as separator because a certain multinational thought it would be more convenient to use ; as separators.

Also agree 💯

We should keep track of all the files which weren't properly parsed and write tests that check the file should be properly parsed, otherwise we might break a successful parsing of another oddly formatted CSV file while fixing this case.

I also agree here, @josihoppe when you look at this issue maybe in a second step you could write a couple tests for this function that parse differently formatted csv files (onw vs. two columns, comma vs. semicolon...) and check that it all works.