Poc/import from dotmny#192
Conversation
- transform.ts: pure functions for reading .mdb tables and transforming Money data (dates, account types, frequencies, currency maps) - migrate.ts: full import pipeline with cleanup, reference data, accounts, transactions, splits, transfers, investments, security prices, exchange rates, scheduled transactions, and balance computation Fixes ON CONFLICT handling for payees and securities to correctly map IDs when duplicates exist. Uses explicit existence check for categories to handle NULL parent_id (where UNIQUE constraint doesn't prevent dupes).
- Load parent .env for POSTGRES_* vars (migration .env only needs MONEY_FILE_PASSWORD and MIGRATION_USER_EMAIL) - Truncate security symbol to 20 chars when falling back to name (securities.symbol is VARCHAR(20)) - Update .env.example to document that Postgres settings are inherited
- Rebuild holdings from investment transactions using the same algorithm as holdings.service.ts rebuildFromTransactions() (78 holdings created) - Deactivate scheduled transactions with next_due_date in the past so they don't show as overdue in Bills & Deposits (2,085 deactivated)
…ions Money's TRN.act field determines investment transaction types. The LOT table proves act=16 closes lots (transfer out / sell) despite its misleading name, while act=15 opens lots (add shares / cost basis). Previous logic inferred action from quantity sign, which was always positive in Money. Holdings now match Money's LOT table exactly.
|
I ran this and it actually worked pretty well! From what I can tell, most bank accounts migrated over completely seamlessly. It even kept my favourite tags! I don't know if it handles open/closed accounts because I had to open all of them to export the QIF. Here's the output: Things I noticed:I had to manually install All loans show up as Mortgages with no transactions in them. The bill payment splits from the source account for the principal show up blank. It should be a TYPE: Transfer and transfer to the loan account It imported every scheduled bill I've ever had since 2003-Oct-15. Not sure why that date, because I have data going back to 1994. I thought they were long gone, but are obviously still in the database. There are now 1844 bills/deposits, where I actually have 20. All are shown as There are a few weird payees called There are some categories that don't appear in MS Money like All securities imported properly. HOWEVER, my investment accounts are a mess. Shares don't line up at all. Can't make any sense of how it arrived at the numbers. One shows as having negative transactions for some reason. Overall, its quite promising. I think the loan issue should be easy to rectify, but I have no idea what's going so wrong with investments. |
|
Oh yes, mortgage is on my todo list. I haven't leant into that yet, as I switched to using liability accounts a decade ago (far more flexible). But I should make sure my historical data is right. Interesting about the bills and deposits. I haven't checked the data quality in my database yet. That's interesting about the accounts. Mine carried over almost perfectly. There is issues with things like the cost basis calculation (for investments in a different currency to your base currency, Money seems to store the base value in the base currency as the exchange rate was at the time). There are some subtle issues with determining buys versus sells that it might have tripped up on. What version of MS Money was your data file from? |
|
I'm using the Sunset Edition. |
|
I've tested this with MSMoney 2001 and after a little help from Claude it seemed to work. I was initially unable to get the backend to start, which Claude identified as: Migration 056_monte_carlo_scenarios.sql tries to create a trigger that already exists in your database, This happened because the migration was partially applied at some point. Fix: manually mark the migration as done: Connect to your postgres container and mark the migration as already applied: Once inside psql, check how migrations are tracked: Look for a table like migrations, schema_migrations, or migration_history, then check what's in it: Then insert the missing record (the exact syntax depends on the table structure, but likely something like): Exit psql with \q, then restart the backend: I then copied my MSMoney file over to the LXC, renamed it to source.mny and ran the migration tool. This failed I was missing libatomic.so.1 It got further, but was missing pg I then had a few Postgres issues, mainly making sure the .env file was set up correctly with the docker IP address,
got further, but failed on table BILL. Claude walked me through checking the tables and modifying the scripts:
First fix — line 347, change: to: Second fix — line 726, same change: Then: Add this function right after the closing } of readTable: Then: Find the import line at the top that imports from transform.ts and add tableExists to it. It probably looks like:
and got: Deleting existing data... Scheduled transactions came across as 0, which is expected since Money 2001 didn't have a separate BILL table. |
3 participants
This is a proof of concept - not ready to merge
Initial attempt at a migration pipeline. Put
source.mnyinmigration, and runmigrate.sh. It should take care of the rest.This successfully imported my MS Money 2004 file, but I haven't done a full reconciliation to make sure all is correct. It looks very close though.
Submitting this for feedback and learning - I do not recommend merging it yet. I also have no idea how it will behave with .mny files from different versions.