Duplicate keys in `newvars` dictionary in the `preprocess_puf` function in `puf.py`

[martinholmer]

When running "make lint" I get among other warnings this one:

tmd/datasets/puf.py:133:14: W0109: Duplicate key 'early_withdrawal_penalty' in dictionary (duplicate-key)

Looking more closely, this looks like a bug:

(base) datasets> grep -n early_withdrawal_penalty puf.py
145:        "early_withdrawal_penalty": puf.E03400,
189:        "early_withdrawal_penalty": puf.E09900,
219:    "early_withdrawal_penalty",

The key-value pairs on lines 145 and 189 are in the same dictionary

@donboyd5, which E value is the correct one for early_withdrawal_penalty?

[martinholmer]

From the Tax-Calculator records_variables.json file, we have:

       "e03400": {
         "type": "float",
         "desc": "Penalty on early withdrawal of savings",
         "form": {"2013-2016": "1040 line 30"},
         "availability": "taxdata_puf, taxmicrodata_tmd"
       },

and

       "e09900": {
         "type": "float",
         "desc": "Penalty tax on qualified retirement plans",
         "form": {"2013-2013": "1040 line 58",
                  "2014-2016": "1040 line 59"},
         "availability": "taxdata_puf, taxmicrodata_tmd"
       },

[martinholmer]

Further investigation shows that this "make lint" warning is definitely a TMD bug.

The confusion about the meaning of the e03400 and e09900 PUF variables, produces incorrect values in tmd.csv for the e03400 and e09900 variables.

Inspection of the Tax-Calculator calcfunctions.py code shows that e03400 is an above-the-line deduction that reduces AGI, while e09900 is a tax. They are conceptually different and affect tax results in different ways.

Tabulation of the puf.csv raw 2015 data show these variables have different values:

> awk -F, 'NR==1{for(i=1;i<=NF;i++)print i,$i}' puf.csv | grep -e e034 -e e099
47 e03400
56 e09900
> awk -F, 'BEGIN{t3=0;t9=0}NR>1{t3+=$47;t9+=$56}END{print t3,t9}' puf.csv 
1933476 15246992

Tabulation of the tmd.csv raw 2021 data show these variables have the same values:

> awk -F, 'NR==1{for(i=1;i<=NF;i++)print i,$i}' tmd.csv | grep -e e034 -e e099
51 e03400
60 e09900
> awk -F, 'BEGIN{t3=0;t9=0}NR>1{t3+=$51;t9+=$60}END{print t3,t9}' tmd.csv
21863439 21863439

[donboyd5]

Thanks, @martinholmer. Preliminary examination suggests that tmd e03400 is incorrectly given the e09900 "Penalty tax on IRA" value when instead it should have the proper "Forfeited interest penalty" value (these are terms used in the PUF 2015 documentation).

I will investigate further and fix this after merging the reweighing PR.

[martinholmer]

@donboyd5 said in issue #405:

I will investigate further and fix this after merging the reweighing PR.

No, don't do that. Bug is already fixed in PR #408.

[donboyd5]

@martinholmer Thanks!