Beangulp (more generally beancount v3) compatibility

[umangv]

@blais recently wrote (I'm paraphrasing) that Beancount v2 is now deprecated, I'm wondering if there is a comprehensive list of blockers for Beancount v3 compatibility.

From what I can tell, the biggest blocker is probably beangulp support. I haven't tried, but it looks like beanquery also needs some work (#1824).

Is there a plan for making fava compatible with beancount v3? If this hasn't already been discussed (sorry if I missed it), I can see two approaches:

1. add logic in fava to handle the breaking changes between v2 and v3 (this would require being able to handle the case where beancount.ingest cannot be imported). In other words, evolve Fava so that it is backwards compatible with Beancount v2 (I imagine this will have a larger maintenance cost). Or,
2. develop a v3 branch of fava until minimal import/querying functionality works with beancount v3 and then announce that branch as available for broader use.

If there's already a plan (or one of the approaches above gets chosen), I'm happy to contribute code.

Two issues that I immediately see with integrating fava with beangulp are:

1. the "Library only" change from beancount v2's ingest to beangulp. This feels like a more fundamental blocker to adopting beangulp with fava, from my understanding (although I will admit I haven't studied this very closely).
2. the file cache used internally in beangulp. It looks like beangulp is designed to be used by scripts and assumes files will not change during the lifetime of the process, while fava is designed to be a long-running server. Files change during the lifetime of the fava process. One solution here might be to patch a change in beangulp that sets a TTL on beangulp's internal cache or exposes an API for cache invalidation.

[umangv]

I was thinking about this a little more and I think one way out of this might be to embrace the "The user will be expected to write their own script." part of the beangulp paradigm and make the interface between Fava and importers be the script itself, with beangulp.Ingest's command line arguments and flags as the inputs and STDOUT as the output. Then Fava can use subprocess to call the user's script. This means that the beangulp.Ingest CLI is no longer just a helper that makes a human-usable CLI out of a list of beangulp.Importers, but it's part of the contract.

@yagebu, thoughts? If this seems OK, I'm happy to take a stab at implementing this.

[yagebu]

Overall, I think this can be done in version of Fava that is compatible to Beancount v2 and v3 and compatible to v2 style Importers as well as beangulp-style importers. See #1860 for a PR that implements this (beangulp-style importers are not yet supported). Doesn't seem like this adds to much of a maintenance burden, there's just minor breaking differences, beanquery works with both and for importers we'll need some conditional logic (to just wrap ImporterProtocol ones in Adapters most likely).

the "Library only" change from beancount v2's ingest to beangulp. This feels like a more fundamental blocker to adopting beangulp with fava, from my understanding (although I will admit I haven't studied this very closely).

I don't think that the "The user will be expected to write their own script." requires a different approach from what we were doing so far. That seems to mostly apply to the "how do I run my importers from the command line", where the trampoline logic in v2 complicated things maybe. I don't see the need to drop down from Python interfaces to just strings and the command line. Importers are still just Python classes that implement a certain interface.

the file cache used internally in beangulp. It looks like beangulp is designed to be used by scripts and assumes files will not change during the lifetime of the process, while fava is designed to be a long-running server. Files change during the lifetime of the fava process. One solution here might be to patch a change in beangulp that sets a TTL on beangulp's internal cache or exposes an API for cache invalidation.

So the in-memory cache used by the v2 importer compatibility layer has this "problem", the one in the @cache (caching to disk) actually does a stat already as well. We could either monkey-patch this cache to make stat calls and check mtimes when running Fava or I think this might make sense to have in beangulp as well (I guess it's not expected to call this cache in a hot loop, so these calls probably don't hurt.