src.org.python.modules.gc Maven / Gradle / Ivy

Go to download
Show more of this group Show more artifacts with this name
Show all versions of jython-standalone Show documentation
Jython is an implementation of the high-level, dynamic, object-oriented language Python written in 100% Pure Java, and seamlessly integrated with the Java platform. It thus allows you to run Python on any Java platform.
There is a newer version: 2.7.4
Show newest version
package org.python.modules;

import java.util.Set;
import java.util.List;
import java.util.ArrayList;
import java.util.IdentityHashMap;
import java.util.HashSet;
import java.util.concurrent.atomic.AtomicBoolean;
import java.lang.ref.Reference;
import java.lang.ref.ReferenceQueue;
import java.lang.ref.WeakReference;
import java.lang.reflect.Array;
import java.lang.reflect.Field;
import java.lang.reflect.Modifier;

import org.python.core.JyAttribute;
import org.python.core.Py;
import org.python.core.PyList;
import org.python.core.PyObject;
import org.python.core.PyInstance;
import org.python.core.PyString;
import org.python.core.Traverseproc;
import org.python.core.TraverseprocDerived;
import org.python.core.Visitproc;
import org.python.core.Untraversable;
import org.python.core.finalization.FinalizeTrigger;
import org.python.modules._weakref.GlobalRef;

//These imports belong to the out-commented section on MXBean-based
//gc-sync far below. That section is kept to document this failed
//approach and allow easy reproduction of this failure.
//import java.lang.management.*;
//import javax.management.*;
//import javax.management.openmbean.*;

/**
 * 
 * In Jython, the gc-module notably differs from that in CPython.
 * This comes from the different ways Jython and CPython perform
 * garbage-collection. While CPython's garbage collection is based on
 * 
 * reference-counting, Jython is backed by Java's gc, which is
 * based on a
 * 
 * mark-and-sweep-approach.
 * 
 * 
 * This difference becomes most notable if finalizers are involved that perform resurrection.
 * While the resurrected object itself behaves rather similar between Jython and CPython,
 * things are more delicate with objects that are reachable (i.e. strongly referenced)
 * via the resurrected object exclusively.
 * While in CPython such objects do not get their finalizers called, Jython/Java would
 * call all their finalizers. That is because Java detects the whole unreachable subgraph
 * as garbage and thus calls all their finalizers without any chance of direct intervention.
 * CPython instead detects the unreachable object and calls its finalizer, which makes the
 * object reachable again. Then all other objects are reachable from it and CPython does not
 * treat them as garbage and does not call their finalizers at all.
 * This further means that in Jython weak references to such indirectly resurrected objects
 * break, while these persist in CPython.
 * 
 * 
 * As of Jython 2.7, the gc-module offers some options to emulate CPython-behavior.
 * Especially see the flags {@link #PRESERVE_WEAKREFS_ON_RESURRECTION},
 * {@link #DONT_FINALIZE_RESURRECTED_OBJECTS} and {@link #DONT_FINALIZE_CYCLIC_GARBAGE}
 * for this.
 * 
 * 
 * Another difference is that CPython's gc-module offers some debug-features like counting
 * of collected cyclic trash, which are hard to support by Jython. As of Jython 2.7 the
 * introduction of a traverseproc-mechanism (c.f. {@link org.python.core.Traverseproc})
 * made support of these features feasible. As support of these features comes
 * with a significant emulation-cost, one must explicitly tell gc to perform this.
 * To make objects subject to cyclic trash-counting, these objects must be gc-monitored in
 * Jython. See {@link #monitorObject(PyObject)}, {@link #unmonitorObject(PyObject)},
 * {@link #MONITOR_GLOBAL} and {@link #stopMonitoring()} for this.

 * If at least one object is gc-monitored, {@link #collect()} works synchronously in the
 * sense that it blocks until all gc-monitored objects that are garbage actually have been
 * collected and had their finalizers called and completed. {@link #collect()} will report
 * the number of collected objects in the same manner as in CPython, i.e. counts only those
 * that participate in reference cycles. This allows a unified test-implementation across
 * Jython and CPython (which applies to most tests in test_gc.py). If not any object is
 * gc-monitored, {@link #collect()} just delegates to {@link java.lang.System.gc()}, runs
 * asynchronously (i.e. non-blocking) and returns {@link #UNKNOWN_COUNT}.
 * See also {@link #DEBUG_SAVEALL} for a useful gc-debugging feature that is supported by
 * Jython from version 2.7 onwards.
 * 
 * 
 * Implementing all these features in Jython involved a lot of synchronization logic.
 * While care was taken to implement this without using timeouts as far as possible and
 * rely on locks, states and system/hardware independent synchronization techniques,
 * this was not entirely feasible.

 * The aspects that were only feasible using a timeout are waiting for gc to enqueue all
 * collected objects (i.e. weak references to monitored objects that were gc'ed) to the
 * reference queue and waiting for gc to run all PyObject-finalizers.
 * 
 * 
 * Waiting for trash could in theory be strictly synchronized by using {@code MXBean}s, i.e.
 * GarbageCollectionNotificationInfo and related API.
 * However, experiments showed that the arising gc-notifications do not reliably indicate
 * when enqueuing was done for a specific gc-run. We kept the experimental implementation
 * in source-code comments to allow easy reproducibility of this issue. (Note that out-commented
 * code contradicts Jython-styleguide, but this one - however - is needed to document this
 * infeasible approach and is explicitly declared accordingly).

 * But how is sync done now?
 * We insert a sentinel before running gc and wait until this sentinel was collected.
 * Timestamps are taken to give us an idea at which time-scales the gc of the current JVM
 * performs. We then wait until twice the measured time (i.e. duration from call to
 * {@link java.lang.System#gc()} until the sentinel reference was enqueued) has passed after
 * the last reference was enqueued by gc. While this approach is not entirely safe in theory,
 * it passes all tests on various systems and machines we had available for testing so far.
 * We consider it more robust than a fixed-length timeout and regard it the best known feasible
 * compromise to emulate synchronous gc-runs in Java.
 * 
 * 
 * The other timing-based synchronization issue - waiting for finalizers to run - is solved as
 * follows. Since PyObject-finalizers are based on
 * {@link org.python.core.finalization.FinalizeTrigger}s, Jython has full control about
 * these finalization process from a central point. Before such a finalizer runs, it calls
 * {@link #notifyPreFinalization()} and when it is done, it calls
 * {@link #notifyPostFinalization()}. While processing of a finalizer can be of arbitrary
 * duration, it widely holds that Java's gc-thread calls the next finalizer almost
 * instantaneously after the former. That means that a timestamp taken in
 * {@link #notifyPreFinalization()} is usually delayed only few milliseconds
 * - often even reported as 0 milliseconds - after the last taken timestamp in
 * {@link #notifyPostFinalization()} (i.e. that was called by the previous finalizer).
 * Jython's gc-module assumes the end of Java's finalization process if
 * {@link #postFinalizationTimeOut} milliseconds passed after a call of
 * {@link #notifyPostFinalization()} without another call to
 * {@link #notifyPreFinalization()} in that time. The default value of
 * {@link #postFinalizationTimeOut} is {@code 100}, which is far larger than the
 * usual almost-zero duration between finalizer calls.

 * This process can be disturbed by third-party finalizers of non-PyObjects brought
 * into the process by external libraries. If these finalizers are of short duration
 * (which applies to typical finalizers), one can deal with this by adjusting
 * {@link #postFinalizationTimeOut}, which was declared {@code public} for exactly this
 * purpose. However if the external framework causing the issue is Jython-aware, a
 * cleaner solution would be to let its finalizers call {@link #notifyPreFinalization()}
 * and {@link #notifyPostFinalization()} appropriately. In that case these finalizers
 * must not terminate by throwing an exception before {@link #notifyPostFinalization()}
 * was called. This is a strict requirement, since a deadlock can be caused otherwise.

 * 

 * Note that the management API
 * (c.f.
 * com.sun.management.GarbageCollectionNotificationInfo) does not emit any
 * notifications that allow to detect the end of the finalization-phase. So this API
 * provides no alternative to the described technique.
 * 
 * 
 * Usually Java's gc provides hardly any guarantee about its collection-  and finalization-
 * process. It not even guarantees that finalizers are called at all (c.f.
 * http://howtodoinjava.com/2012/10/31/why-not-to-use-finalize-method-in-java).
 * While at least the most common JVM implementations usually do call finalizers
 * reliably under normal conditions, there still is no specific finalization-order guaranteed
 * (one might reasonably expect that this would be related to reference-connection graph
 * topology, but this appears not to be the case).
 * However Jython now offers some functionality to compensate this
 * situation. Via {@link #registerPreFinalizationProcess(Runnable)} and
 * {@link #registerPostFinalizationProcess(Runnable)} and related methods one can now
 * listen to beginning and end of the finalization process. Note that this functionality
 * relies on the technique described in the former paragraph (i.e. based on calls to
 * {@link #notifyPreFinalization()} and {@link #notifyPostFinalization()}) and thus
 * underlies its unsafety, if third-party finalizers are involved. Such finalizers can
 * cause false-positive runs of registered (pre/post)-finalization-processes, so this
 * feature should be used with some care. It is recommended to use it only in such a way
 * that false-positive runs would not cause serious harm, but only some loss in
 * performance or so.
 * 
 */
public class gc {
    /**
     * A constant that can occur as result of {@link #collect()} and
     * indicates an unknown number of collected cyclic trash.
     * It is intentionally not valued -1 as that value is
     * reserved to indicate an error.
     */
    public static final int UNKNOWN_COUNT = -2;
    
    /* Jython-specific gc-flags: */
    /**
     * This flag tells every newly created PyObject to register for
     * gc-monitoring. This allows {@link #collect()} to report the
     * number of collected objects.
     *
     * @see #setJythonGCFlags(short)
     * @see #getJythonGCFlags()
     * @see #addJythonGCFlags(short)
     * @see #removeJythonGCFlags(short)
     */
    public static final short MONITOR_GLOBAL =                    (1<<0);

    /**
     * CPython prior to 3.4 does not finalize cyclic garbage
     * PyObjects, while Jython does this by default. This flag
     * tells Jython's gc to mimic CPython <3.4 behavior (i.e.
     * add such objects to {@code gc.garbage} list instead).
     *
     * @see #setJythonGCFlags(short)
     * @see #getJythonGCFlags()
     * @see #addJythonGCFlags(short)
     * @see #removeJythonGCFlags(short)
     */
    public static final short DONT_FINALIZE_CYCLIC_GARBAGE =      (1<<1);

    /**
     * If a PyObject is resurrected during its finalization
     * process and was weakly referenced, Jython breaks the
     * weak references to the resurrected PyObject by default.
     * In CPython these persist, if the object was indirectly
     * resurrected due to resurrection of its owner.
     * This flag tells Jython's gc to preserve weak references
     * to such resurrected PyObjects.
     * It only works if all involved objects implement the
     * traverseproc mechanism properly (see
     * {@link org.python.core.Traverseproc}).
     * Note that this feature comes with some cost as it can
     * delay garbage collection of some weak referenced objects
     * for several gc cycles if activated. So we recommend to
     * use it only for debugging.
     *
     * @see #setJythonGCFlags(short)
     * @see #getJythonGCFlags()
     * @see #addJythonGCFlags(short)
     * @see #removeJythonGCFlags(short)
     */
    public static final short PRESERVE_WEAKREFS_ON_RESURRECTION = (1<<2);

    /**
     * If in CPython an object is resurrected via its finalizer
     * and contained strong references to other objects, these
     * are also resurrected and not finalized in CPython (as
     * their reference count never drops to zero). In contrast
     * to that, Jython calls finalizers for all objects that
     * were unreachable when gc started (regardless of resurrections
     * and in unpredictable order). This flag emulates CPython
     * behavior in Jython. Note that this emulation comes with a
     * significant cost as it can delay collection of many objects
     * for several gc-cycles. Its main intention is for debugging
     * resurrection-sensitive code.
     *
     * @see #setJythonGCFlags(short)
     * @see #getJythonGCFlags()
     * @see #addJythonGCFlags(short)
     * @see #removeJythonGCFlags(short)
     */
    public static final short DONT_FINALIZE_RESURRECTED_OBJECTS = (1<<3);

    /**
     * 
     * Reflection-based traversion is an inefficient fallback-method to
     * traverse PyObject-subtypes that don't implement
     * {@link org.python.core.Traverseproc} and
     * are not marked as {@link org.python.core.Untraversable}.
     * Such a situation indicates that the programmer was not aware of
     * Jython's traverseproc-mechanism and reflection is used to
     * compensate this.
     * 
     * 
     * This flag allows to inhibit reflection-based traversion. If it is
     * activated, objects that don't implement
     * {@link org.python.core.Traverseproc}
     * are always treated as if they were marked as
     * {@link org.python.core.Untraversable}.
     * 
     * 
     * Note that reflection-based traversion fallback is performed by
     * default. Further note that Jython emits warning-messages if
     * reflection-based traversion occurs or if an object is encountered
     * that neither implements {@link org.python.core.Traverseproc}
     * nor is marked as {@link org.python.core.Untraversable} (even if
     * reflection-based traversion is inhibited). See
     * {@link #SUPPRESS_TRAVERSE_BY_REFLECTION_WARNING} and
     * {@link #INSTANCE_TRAVERSE_BY_REFLECTION_WARNING} to control
     * these warning-messages.
     * 
     *
     * @see #setJythonGCFlags(short)
     * @see #getJythonGCFlags()
     * @see #addJythonGCFlags(short)
     * @see #removeJythonGCFlags(short)
     * @see #SUPPRESS_TRAVERSE_BY_REFLECTION_WARNING
     * @see #INSTANCE_TRAVERSE_BY_REFLECTION_WARNING
     */
    public static final short DONT_TRAVERSE_BY_REFLECTION =       (1<<4);

    /**
     * 
     * If this flag is not set, gc warns whenever an object would be subject to
     * reflection-based traversion.
     * Note that if this flag is not set, the warning will occur even if
     * reflection-based traversion is not active. The purpose of this behavior is
     * to identify objects that don't properly support the traverseproc-mechanism,
     * i.e. instances of PyObject-subclasses that neither implement
     * {@link org.python.core.Traverseproc},
     * nor are annotated with the {@link org.python.core.Untraversable}-annotation.
     * 
     * 
     * A SUPPRESS-flag was chosen rather than a WARN-flag, so that warning is the
     * default behavior - the user must actively set this flag in order to not to
     * be warned.
     * This is because in an ideal implementation reflection-based traversion never
     * occurs; it is only an inefficient fallback.
     * 
     *
     * @see #setJythonGCFlags(short)
     * @see #getJythonGCFlags()
     * @see #addJythonGCFlags(short)
     * @see #removeJythonGCFlags(short)
     * @see #INSTANCE_TRAVERSE_BY_REFLECTION_WARNING
     */
    public static final short SUPPRESS_TRAVERSE_BY_REFLECTION_WARNING =    (1<<5);

    /**
     * Makes gc emit reflection-based traversion warning for every traversed
     * object instead of only once per class.
     * A potential reflection-based traversion occurs whenever an object is
     * traversed that neither implements {@link org.python.core.Traverseproc},
     * nor is annotated with the {@link org.python.core.Untraversable}-annotation.
     *
     * @see #setJythonGCFlags(short)
     * @see #getJythonGCFlags()
     * @see #addJythonGCFlags(short)
     * @see #removeJythonGCFlags(short)
     * @see #SUPPRESS_TRAVERSE_BY_REFLECTION_WARNING
     */
    public static final short INSTANCE_TRAVERSE_BY_REFLECTION_WARNING =    (1<<6);

    /**
     * In Jython one usually uses {@code Py.writeDebug} for debugging output.
     * However that method is only verbose if an appropriate verbose-level
     * was set. In CPython it is enough to set gc-{@code DEBUG} flags to get
     * gc-messages, no matter what overall verbose level is selected.
     * This flag tells Jython to use {@code Py.writeDebug} for debugging output.
     * If it is not set (default-case), gc-debugging output (if gc-{@code VERBOSE}
     * or -{@code DEBUG} flags are set) is directly written to {@code System.err}.
     *
     * @see #setJythonGCFlags(short)
     * @see #getJythonGCFlags()
     * @see #addJythonGCFlags(short)
     * @see #removeJythonGCFlags(short)
     */
    public static final short USE_PY_WRITE_DEBUG = (1<<7);

    /**
     * Enables collection-related verbose-output.
     * 
     * @see #setJythonGCFlags(short)
     * @see #getJythonGCFlags()
     * @see #addJythonGCFlags(short)
     * @see #removeJythonGCFlags(short)
     */
    public static final short VERBOSE_COLLECT =  (1<<8);

    /**
     * Enables weakref-related verbose-output.
     *
     * @see #setJythonGCFlags(short)
     * @see #getJythonGCFlags()
     * @see #addJythonGCFlags(short)
     * @see #removeJythonGCFlags(short)
     */
    public static final short VERBOSE_WEAKREF =  (1<<9);

    /**
     * Enables delayed finalization related verbose-output.
     *
     * @see #setJythonGCFlags(short)
     * @see #getJythonGCFlags()
     * @see #addJythonGCFlags(short)
     * @see #removeJythonGCFlags(short)
     */
    public static final short VERBOSE_DELAYED =  (1<<10);

    /**
     * Enables finalization-related verbose-output.
     *
     * @see #setJythonGCFlags(short)
     * @see #getJythonGCFlags()
     * @see #addJythonGCFlags(short)
     * @see #removeJythonGCFlags(short)
     */
    public static final short VERBOSE_FINALIZE = (1<<11);

    /**
     * Bit-combination of the flags {@link #VERBOSE_COLLECT},
     * {@link #VERBOSE_WEAKREF}, {@link #VERBOSE_DELAYED},
     * {@link #VERBOSE_FINALIZE}.
     *
     * @see #setJythonGCFlags(short)
     * @see #getJythonGCFlags()
     * @see #addJythonGCFlags(short)
     * @see #removeJythonGCFlags(short)
     */
    public static final short VERBOSE =
            VERBOSE_COLLECT | VERBOSE_WEAKREF | VERBOSE_DELAYED | VERBOSE_FINALIZE;

    /* set for debugging information */
    /**
     * print collection statistics
     * (in Jython scoped on monitored objects)
     *
     * @see #set_debug(int)
     * @see #get_debug()
     */
    public static final int DEBUG_STATS         = (1<<0);

    /**
     * print collectable objects
     * (in Jython scoped on monitored objects)
     *
     * @see #set_debug(int)
     * @see #get_debug()
     */
    public static final int DEBUG_COLLECTABLE   = (1<<1);

    /**
     * print uncollectable objects
     * (in Jython scoped on monitored objects)
     *
     * @see #set_debug(int)
     * @see #get_debug()
     */
    public static final int DEBUG_UNCOLLECTABLE = (1<<2);

    /**
     * print instances
     * (in Jython scoped on monitored objects)
     *
     * @see #set_debug(int)
     * @see #get_debug()
     */
    public static final int DEBUG_INSTANCES     = (1<<3);

    /**
     * print other objects
     * (in Jython scoped on monitored objects)
     *
     * @see #set_debug(int)
     * @see #get_debug()
     */
    public static final int DEBUG_OBJECTS       = (1<<4);

    /**
     * save all garbage in gc.garbage
     * (in Jython scoped on monitored objects)
     *
     * @see #set_debug(int)
     * @see #get_debug()
     */
    public static final int DEBUG_SAVEALL       = (1<<5);

    /**
     * Bit-combination of the flags {@link #DEBUG_COLLECTABLE},
     * {@link #DEBUG_UNCOLLECTABLE}, {@link #DEBUG_INSTANCES},
     * {@link #DEBUG_OBJECTS}, {@link #DEBUG_SAVEALL}.
     *
     * @see #set_debug(int)
     * @see #get_debug()
     */
    public static final int DEBUG_LEAK = DEBUG_COLLECTABLE |
                                         DEBUG_UNCOLLECTABLE |
                                         DEBUG_INSTANCES |
                                         DEBUG_OBJECTS |
                                         DEBUG_SAVEALL;

    private static short gcFlags = 0;
    private static int debugFlags = 0;
    private static boolean monitorNonTraversable = false;
    private static boolean waitingForFinalizers = false;
    private static AtomicBoolean gcRunning = new AtomicBoolean(false);
    private static HashSet monitoredObjects;
    private static HashSet> reflectionWarnedClasses;
    private static ReferenceQueue