org.python.modules.gc Maven / Gradle / Ivy

Show more of this group Show more artifacts with this name
Show all versions of jython-slim Show documentation
Jython is an implementation of the high-level, dynamic, object-oriented language Python written in 100% Pure Java, and seamlessly integrated with the Java platform. It thus allows you to run Python on any Java platform.
The newest version!
package org.python.modules;

import java.util.Set;
import java.util.List;
import java.util.ArrayList;
import java.util.IdentityHashMap;
import java.util.HashSet;
import java.util.concurrent.atomic.AtomicBoolean;
import java.lang.ref.Reference;
import java.lang.ref.ReferenceQueue;
import java.lang.ref.WeakReference;
import java.lang.reflect.Array;
import java.lang.reflect.Field;
import java.lang.reflect.Modifier;

import org.python.core.JyAttribute;
import org.python.core.Py;
import org.python.core.PyException;
import org.python.core.PyList;
import org.python.core.PyObject;
import org.python.core.PyInstance;
import org.python.core.PyString;
import org.python.core.Traverseproc;
import org.python.core.TraverseprocDerived;
import org.python.core.Visitproc;
import org.python.core.Untraversable;
import org.python.core.finalization.FinalizeTrigger;
import org.python.modules._weakref.GlobalRef;
import org.python.modules._weakref.ReferenceBackend;

//These imports belong to the out commented section on MXBean-based
//gc sync far below. That section is kept to document this failed
//approach and allow easy reproduction of this failure.
//import java.lang.management.*;
//import javax.management.*;
//import javax.management.openmbean.*;

/**
 * In Jython, the gc module notably differs from that in CPython. This comes from the different ways
 * Jython and CPython perform garbage collection. While CPython's garbage collection is based on
 *  reference
 * counting, Jython is backed by Java's gc, which is based on a
 *  mark-and-sweep
 * approach.
 * 
 * This difference becomes most notable if finalizers are involved that perform resurrection. While
 * the resurrected object itself behaves rather similar between Jython and CPython, things are more
 * delicate with objects that are reachable (i.e. strongly referenced) via the resurrected object
 * exclusively. While in CPython such objects do not get their finalizers called, Jython/Java would
 * call all their finalizers. That is because Java detects the whole unreachable subgraph as garbage
 * and thus calls all their finalizers without any chance of direct intervention. CPython instead
 * detects the unreachable object and calls its finalizer, which makes the object reachable again.
 * Then all other objects are reachable from it and CPython does not treat them as garbage and does
 * not call their finalizers at all. This further means that in Jython weak references to such
 * indirectly resurrected objects break, while these persist in CPython.
 * 

 * As of Jython 2.7, the gc module offers some options to emulate CPython behavior. Especially see
 * the flags {@link #PRESERVE_WEAKREFS_ON_RESURRECTION}, {@link #DONT_FINALIZE_RESURRECTED_OBJECTS}
 * and {@link #DONT_FINALIZE_CYCLIC_GARBAGE} for this.
 * 

 * Another difference is that CPython's gc module offers some debug features like counting of
 * collected cyclic trash, which are hard to support by Jython. As of Jython 2.7 the introduction of
 * a traverseproc mechanism (c.f. {@link org.python.core.Traverseproc}) made support of these
 * features feasible. As support of these features comes with a significant emulation cost, one must
 * explicitly tell gc to perform this. To make objects subject to cyclic trash counting, these
 * objects must be gc-monitored in Jython. See {@link #monitorObject(PyObject)},
 * {@link #unmonitorObject(PyObject)}, {@link #MONITOR_GLOBAL} and {@link #stopMonitoring()} for
 * this.
 * 

 * If at least one object is gc-monitored, {@link #collect()} works synchronously in the sense that
 * it blocks until all gc-monitored objects that are garbage actually have been collected and had
 * their finalizers called and completed. {@link #collect()} will report the number of collected
 * objects in the same manner as in CPython, i.e. counts only those that participate in reference
 * cycles. This allows a unified test implementation across Jython and CPython (which applies to
 * most tests in test_gc.py). If not any object is gc-monitored, {@link #collect()} just delegates
 * to {@link java.lang.System#gc()}, runs asynchronously (i.e. non-blocking) and returns
 * {@link #UNKNOWN_COUNT}. See also {@link #DEBUG_SAVEALL} for a useful gc debugging feature that is
 * supported by Jython from version 2.7 onwards.
 * 

 * Implementing all these features in Jython involved a lot of synchronization logic. While care was
 * taken to implement this without using timeouts as far as possible and rely on locks, states and
 * system/hardware independent synchronization techniques, this was not entirely feasible.

 * The aspects that were only feasible using a timeout are waiting for gc to enqueue all collected
 * objects (i.e. weak references to monitored objects that were gc'ed) to the reference queue and
 * waiting for gc to run all PyObject finalizers.
 * 

 * Waiting for trash could in theory be strictly synchronized by using {@code MXBean}s, i.e.
 * GarbageCollectionNotificationInfo and related API. However, experiments
 * showed that the arising gc notifications do not reliably indicate when enqueuing was done for a
 * specific gc run. We kept the experimental implementation in source code comments to allow easy
 * reproducibility of this issue. (Note that out commented code contradicts Jython styleguide, but
 * this one - however - is needed to document this infeasible approach and is explicitly declared
 * accordingly).
 * 

 * But how is sync done now? We insert a sentinel before running gc and wait until this
 * sentinel was collected. Timestamps are taken to give us an idea at which time scales the gc of
 * the current JVM performs. We then wait until twice the measured time (i.e. duration from call to
 * {@link java.lang.System#gc()} until the sentinel reference was enqueued) has passed after the
 * last reference was enqueued by gc. While this approach is not entirely safe in theory, it passes
 * all tests on various systems and machines we had available for testing so far. We consider it
 * more robust than a fixed-length timeout and regard it the best known feasible compromise to
 * emulate synchronous gc runs in Java.
 * 

 * The other timing-based synchronization issue - waiting for finalizers to run - is solved as
 * follows. Since PyObject finalizers are based on
 * {@link org.python.core.finalization.FinalizeTrigger}s, Jython has full control about these
 * finalization process from a central point. Before such a finalizer runs, it calls
 * {@link #notifyPreFinalization()} and when it is done, it calls {@link #notifyPostFinalization()}.
 * While processing of a finalizer can be of arbitrary duration, it widely holds that Java's gc
 * thread calls the next finalizer almost instantaneously after the former. That means that a
 * timestamp taken in {@link #notifyPreFinalization()} is usually delayed only few milliseconds -
 * often even reported as 0 milliseconds - after the last taken timestamp in
 * {@link #notifyPostFinalization()} (i.e. that was called by the previous finalizer). Jython's gc
 * module assumes the end of Java's finalization process if {@link #postFinalizationTimeOut}
 * milliseconds passed after a call of {@link #notifyPostFinalization()} without another call to
 * {@link #notifyPreFinalization()} in that time. The default value of
 * {@link #postFinalizationTimeOut} is {@code 100}, which is far larger than the usual almost-zero
 * duration between finalizer calls.

 * This process can be disturbed by third-party finalizers of non-PyObjects brought into the process
 * by external libraries. If these finalizers are of short duration (which applies to typical
 * finalizers), one can deal with this by adjusting {@link #postFinalizationTimeOut}, which was
 * declared {@code public} for exactly this purpose. However if the external framework causing the
 * issue is Jython aware, a cleaner solution would be to let its finalizers call
 * {@link #notifyPreFinalization()} and {@link #notifyPostFinalization()} appropriately. In that
 * case these finalizers must not terminate by throwing an exception before
 * {@link #notifyPostFinalization()} was called. This is a strict requirement, since a deadlock can
 * be caused otherwise.
 * 

 * Note that the management API (c.f. com.sun.management.GarbageCollectionNotificationInfo) does not emit any
 * notifications that allow to detect the end of the finalization phase. So this API provides no
 * alternative to the described technique.
 * 

 * Usually Java's gc provides hardly any guarantee about its collection and finalization process. It
 * not even guarantees that finalizers are called at all (c.f.
 * http://howtodoinjava.com/2012/10/31/why-not-to-use-finalize-method-in-java). While
 * at least the most common JVM implementations usually do call finalizers reliably under
 * normal conditions, there still is no specific finalization order guaranteed (one might reasonably
 * expect that this would be related to reference connection graph topology, but this appears not to
 * be the case). However Jython now offers some functionality to compensate this situation. Via
 * {@link #registerPreFinalizationProcess(Runnable)} and
 * {@link #registerPostFinalizationProcess(Runnable)} and related methods one can now listen to
 * beginning and end of the finalization process. Note that this functionality relies on the
 * technique described in the former paragraph (i.e. based on calls to
 * {@link #notifyPreFinalization()} and {@link #notifyPostFinalization()}) and thus underlies its
 * unsafety, if third-party finalizers are involved. Such finalizers can cause false-positive runs
 * of registered (pre/post) finalization processes, so this feature should be used with some care.
 * It is recommended to use it only in such a way that false-positive runs would not cause serious
 * harm, but only some loss in performance or so.
 */
public class gc {
    /**
     * A constant that can occur as result of {@link #collect()} and
     * indicates an unknown number of collected cyclic trash.
     * It is intentionally not valued -1 as that value is
     * reserved to indicate an error.
     */
    public static final int UNKNOWN_COUNT = -2;

    /* Jython-specific gc flags: */
    /**
     * This flag tells every newly created PyObject to register for
     * gc monitoring. This allows {@link #collect()} to report the
     * number of collected objects.
     *
     * @see #setJythonGCFlags(short)
     * @see #getJythonGCFlags()
     * @see #addJythonGCFlags(short)
     * @see #removeJythonGCFlags(short)
     */
    public static final short MONITOR_GLOBAL =                    (1<<0);

    /**
     * CPython prior to 3.4 does not finalize cyclic garbage
     * PyObjects, while Jython does this by default. This flag
     * tells Jython's gc to mimic CPython <3.4 behavior (i.e.
     * add such objects to {@code gc.garbage} list instead).
     *
     * @see #setJythonGCFlags(short)
     * @see #getJythonGCFlags()
     * @see #addJythonGCFlags(short)
     * @see #removeJythonGCFlags(short)
     */
    public static final short DONT_FINALIZE_CYCLIC_GARBAGE =      (1<<1);

    /**
     * If a PyObject is resurrected during its finalization
     * process and was weakly referenced, Jython breaks the
     * weak references to the resurrected PyObject by default.
     * In CPython these persist, if the object was indirectly
     * resurrected due to resurrection of its owner.
     * This flag tells Jython's gc to preserve weak references
     * to such resurrected PyObjects.
     * It only works if all involved objects implement the
     * traverseproc mechanism properly (see
     * {@link org.python.core.Traverseproc}).
     * Note that this feature comes with some cost as it can
     * delay garbage collection of some weak referenced objects
     * for several gc cycles if activated. So we recommend to
     * use it only for debugging.
     *
     * @see #setJythonGCFlags(short)
     * @see #getJythonGCFlags()
     * @see #addJythonGCFlags(short)
     * @see #removeJythonGCFlags(short)
     */
    public static final short PRESERVE_WEAKREFS_ON_RESURRECTION = (1<<2);

    /**
     * If in CPython an object is resurrected via its finalizer
     * and contained strong references to other objects, these
     * are also resurrected and not finalized in CPython (as
     * their reference count never drops to zero). In contrast
     * to that, Jython calls finalizers for all objects that
     * were unreachable when gc started (regardless of resurrections
     * and in unpredictable order). This flag emulates CPython
     * behavior in Jython. Note that this emulation comes with a
     * significant cost as it can delay collection of many objects
     * for several gc cycles. Its main intention is for debugging
     * resurrection-sensitive code.
     *
     * @see #setJythonGCFlags(short)
     * @see #getJythonGCFlags()
     * @see #addJythonGCFlags(short)
     * @see #removeJythonGCFlags(short)
     */
    public static final short DONT_FINALIZE_RESURRECTED_OBJECTS = (1<<3);

    public static final short FORCE_DELAYED_FINALIZATION = (1<<4);
    public static final short FORCE_DELAYED_WEAKREF_CALLBACKS = (1<<5);

    /**
     * 

     * Reflection-based traversal is an inefficient fallback method to
     * traverse PyObject subtypes that don't implement
     * {@link org.python.core.Traverseproc} and
     * are not marked as {@link org.python.core.Untraversable}.
     * Such a situation indicates that the programmer was not aware of
     * Jython's traverseproc mechanism and reflection is used to
     * compensate this.
     * 
     * 
     * This flag allows to inhibit reflection-based traversal. If it is
     * activated, objects that don't implement
     * {@link org.python.core.Traverseproc}
     * are always treated as if they were marked as
     * {@link org.python.core.Untraversable}.
     * 
     * 
     * Note that reflection-based traversal fallback is performed by
     * default. Further note that Jython emits warning messages if
     * reflection-based traversal occurs or if an object is encountered
     * that neither implements {@link org.python.core.Traverseproc}
     * nor is marked as {@link org.python.core.Untraversable} (even if
     * reflection-based traversal is inhibited). See
     * {@link #SUPPRESS_TRAVERSE_BY_REFLECTION_WARNING} and
     * {@link #INSTANCE_TRAVERSE_BY_REFLECTION_WARNING} to control
     * these warning messages.
     * 
     *
     * @see #setJythonGCFlags(short)
     * @see #getJythonGCFlags()
     * @see #addJythonGCFlags(short)
     * @see #removeJythonGCFlags(short)
     * @see #SUPPRESS_TRAVERSE_BY_REFLECTION_WARNING
     * @see #INSTANCE_TRAVERSE_BY_REFLECTION_WARNING
     */
    public static final short DONT_TRAVERSE_BY_REFLECTION =       (1<<6);

    /**
     * 
     * If this flag is not set, gc warns whenever an object would be subject to
     * reflection-based traversal.
     * Note that if this flag is not set, the warning will occur even if
     * reflection-based traversal is not active. The purpose of this behavior is
     * to identify objects that don't properly support the traverseproc mechanism,
     * i.e. instances of PyObject subclasses that neither implement
     * {@link org.python.core.Traverseproc},
     * nor are annotated with the {@link org.python.core.Untraversable} annotation.
     * 
     * 
     * A SUPPRESS flag was chosen rather than a WARN flag, so that warning is the
     * default behavior - the user must actively set this flag in order to not to
     * be warned.
     * This is because in an ideal implementation reflection-based traversal never
     * occurs; it is only an inefficient fallback.
     * 
     *
     * @see #setJythonGCFlags(short)
     * @see #getJythonGCFlags()
     * @see #addJythonGCFlags(short)
     * @see #removeJythonGCFlags(short)
     * @see #INSTANCE_TRAVERSE_BY_REFLECTION_WARNING
     */
    public static final short SUPPRESS_TRAVERSE_BY_REFLECTION_WARNING =    (1<<7);

    /**
     * Makes gc emit reflection-based traversal warning for every traversed
     * object instead of only once per class.
     * A potential reflection-based traversal occurs whenever an object is
     * traversed that neither implements {@link org.python.core.Traverseproc},
     * nor is annotated with the {@link org.python.core.Untraversable} annotation.
     *
     * @see #setJythonGCFlags(short)
     * @see #getJythonGCFlags()
     * @see #addJythonGCFlags(short)
     * @see #removeJythonGCFlags(short)
     * @see #SUPPRESS_TRAVERSE_BY_REFLECTION_WARNING
     */
    public static final short INSTANCE_TRAVERSE_BY_REFLECTION_WARNING =    (1<<8);

    /**
     * In Jython one usually uses {@code Py.writeDebug} for debugging output.
     * However that method is only verbose if an appropriate verbose level
     * was set. In CPython it is enough to set gc {@code DEBUG} flags to get
     * gc messages, no matter what overall verbose level is selected.
     * This flag tells Jython to use {@code Py.writeDebug} for debugging output.
     * If it is not set (default case), gc debugging output (if gc {@code VERBOSE}
     * or {@code DEBUG} flags are set) is directly written to {@code System.err}.
     *
     * @see #setJythonGCFlags(short)
     * @see #getJythonGCFlags()
     * @see #addJythonGCFlags(short)
     * @see #removeJythonGCFlags(short)
     */
    public static final short USE_PY_WRITE_DEBUG = (1<<9);

    /**
     * Enables collection-related verbose output.
     *
     * @see #setJythonGCFlags(short)
     * @see #getJythonGCFlags()
     * @see #addJythonGCFlags(short)
     * @see #removeJythonGCFlags(short)
     */
    public static final short VERBOSE_COLLECT =  (1<<10);

    /**
     * Enables weakref-related verbose output.
     *
     * @see #setJythonGCFlags(short)
     * @see #getJythonGCFlags()
     * @see #addJythonGCFlags(short)
     * @see #removeJythonGCFlags(short)
     */
    public static final short VERBOSE_WEAKREF =  (1<<11);

    /**
     * Enables delayed finalization related verbose output.
     *
     * @see #setJythonGCFlags(short)
     * @see #getJythonGCFlags()
     * @see #addJythonGCFlags(short)
     * @see #removeJythonGCFlags(short)
     */
    public static final short VERBOSE_DELAYED =  (1<<12);

    /**
     * Enables finalization-related verbose output.
     *
     * @see #setJythonGCFlags(short)
     * @see #getJythonGCFlags()
     * @see #addJythonGCFlags(short)
     * @see #removeJythonGCFlags(short)
     */
    public static final short VERBOSE_FINALIZE = (1<<13);

    /**
     * Bit combination of the flags {@link #VERBOSE_COLLECT},
     * {@link #VERBOSE_WEAKREF}, {@link #VERBOSE_DELAYED},
     * {@link #VERBOSE_FINALIZE}.
     *
     * @see #setJythonGCFlags(short)
     * @see #getJythonGCFlags()
     * @see #addJythonGCFlags(short)
     * @see #removeJythonGCFlags(short)
     */
    public static final short VERBOSE =
            VERBOSE_COLLECT | VERBOSE_WEAKREF | VERBOSE_DELAYED | VERBOSE_FINALIZE;

    /* set for debugging information */
    /**
     * print collection statistics
     * (in Jython scoped on monitored objects)
     *
     * @see #set_debug(int)
     * @see #get_debug()
     */
    public static final int DEBUG_STATS         = (1<<0);

    /**
     * print collectable objects
     * (in Jython scoped on monitored objects)
     *
     * @see #set_debug(int)
     * @see #get_debug()
     */
    public static final int DEBUG_COLLECTABLE   = (1<<1);

    /**
     * print uncollectable objects
     * (in Jython scoped on monitored objects)
     *
     * @see #set_debug(int)
     * @see #get_debug()
     */
    public static final int DEBUG_UNCOLLECTABLE = (1<<2);

    /**
     * print instances
     * (in Jython scoped on monitored objects)
     *
     * @see #set_debug(int)
     * @see #get_debug()
     */
    public static final int DEBUG_INSTANCES     = (1<<3);

    /**
     * print other objects
     * (in Jython scoped on monitored objects)
     *
     * @see #set_debug(int)
     * @see #get_debug()
     */
    public static final int DEBUG_OBJECTS       = (1<<4);

    /**
     * save all garbage in gc.garbage
     * (in Jython scoped on monitored objects)
     *
     * @see #set_debug(int)
     * @see #get_debug()
     */
    public static final int DEBUG_SAVEALL       = (1<<5);

    /**
     * Bit combination of the flags {@link #DEBUG_COLLECTABLE},
     * {@link #DEBUG_UNCOLLECTABLE}, {@link #DEBUG_INSTANCES},
     * {@link #DEBUG_OBJECTS}, {@link #DEBUG_SAVEALL}.
     *
     * @see #set_debug(int)
     * @see #get_debug()
     */
    public static final int DEBUG_LEAK = DEBUG_COLLECTABLE |
                                         DEBUG_UNCOLLECTABLE |
                                         DEBUG_INSTANCES |
                                         DEBUG_OBJECTS |
                                         DEBUG_SAVEALL;

    private static short gcFlags = DONT_TRAVERSE_BY_REFLECTION;
    private static int debugFlags = 0;
    private static boolean monitorNonTraversable = false;
    private static boolean waitingForFinalizers = false;
    private static final AtomicBoolean gcRunning = new AtomicBoolean(false);
    private static final Set monitoredObjects = new HashSet<>();
    private static HashSet> reflectionWarnedClasses;
    private static ReferenceQueue