Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add IPython-specific session dump and load capability #103

Open
wants to merge 2 commits into
base: master
Choose a base branch
from
Open

Add IPython-specific session dump and load capability #103

wants to merge 2 commits into from

Conversation

abrasive
Copy link
Contributor

IPython imports a bunch of stuff into the user namespace in Pylab and Matplotlib modes (presumably amongst others). These implicit imports are hidden from the user in commands like %who.

This patch is in two parts. The first modifies dump_session, adding a filtering capability to the new byref mechanism. A flag is also added to silently discard any byref object which cannot be found in imported modules.

The second part adds IPython dump and load functions. The dumper marks any name in the IPython hidden namespace for byref pickling. This includes the implicit imports, as well as the hidden history variables. The history variables and any dynamically generated classes end up discarded by the byref mechanism. The hidden namespace list is saved in the dumped session.
The loader loads the session in the usual way, and then fixes up the hidden namespace again according to the information saved in the dump. This ensures that a subsequent dump and load will continue to work, as well as ensuring consistent behaviour of %who and friends.

byref_filter allows the caller to supply a function that determines
whether a given name should be pickled byref. This permits white- or
black-listing of imported items.

byref_discard_missing specifies whether names marked by the byref_filter
which cannot be found should be silently discarded. This allows
dynamically-generated items to be specifically excluded.
These use the byref filtering mechanism to avoid pickling hidden items
from the namespace. In Pylab or Matplotlib mode, this covers the
implicit "from x import *". Items that cannot be found are silently
discarded, which takes care of dynamically generated classes such as
those from GObject which get thoughtlessly plunked into __main__ by the
GTK backend.

This also restores the hidden namespace after loading, ensuring that
IPython's special handling of them in %who and so forth remains
effective.
@mmckerns
Copy link
Member

@abrasive: I barely looked at this, but as a start: can you not assume that the import IPython works? Also, consider both use and speed in 2.x and 3.x (I didn't look closely for specific places this is applicable). Lastly, I don't like dump_ipython and load_ipython -- dill shouldn't have a separate sessiondump and load for IPython sessions, it should be integrated into the existing dump_session and load_session.

@abrasive
Copy link
Contributor Author

I didn't want to assume that all callers from IPython actually want to use IPython-specific dump and load wrappers. I'll integrate it into dump_session and load_session and remove the import when I have some time.

@mmckerns
Copy link
Member

dump_session has always been not quite good enough for use inIPython, so I appreciate the PR's. The integration of non-standard library solutions into dill is always a tough call…

@mmckerns
Copy link
Member

mmckerns commented Jun 2, 2015

After giving this some more thought, and a better look, I have some updated comments. First, I don't believe dump_ipython and load_ipython don't belong in dill, they are probably better added to ipython -- since I expect this to be a point of contention, please split it off into a separate PR. As far as the other changes (filter and discard), I have no qualms with going in. So once you rename the flags to something more concise (let's pick some names that we agree on), I'll pull the remaining changes. Also, consider that byref, filter, and discard are actually one or two fields instead of three… if they can be condensed into less flags they should be. A typical indication of the need for a collapse is when certain flags only make sense when another flag is set to a particular value -- this is definitely the case here, so I expect that you could collapse them.

@leogama
Copy link
Contributor

leogama commented May 8, 2022

@mmckerns I'm working on this issue with a different kind of solution. Code is growing. What do you think of splitting "session" related code to a submodule? Beyond its size, it also lays down in the way when navigating between imports and definitions and the Pickler and Unpickler code in _dill.py.

@mmckerns
Copy link
Member

mmckerns commented May 9, 2022

I would think that the session related code is pretty tied up with the other code, and hard to separate. It would be interesting to see this attempted...

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants