The currently deployed model is braai that uses deep learning and is published here: https://arxiv.org/abs/1907.11259 with the associated GitHub including notebooks.

• Automated Data Contamination [ Charlotte ] * Find contamination using clustering.
• Active Learning to Improve Training Data Selection [ Sara ] * Use active learning to discover potential batches of boguses (and reals, alternatively)

• finish the work with the alternate test set, and getting stats on that.
• make sure imputation is happening correctly
• get Tomas's other plots integrated
• KL divergence code, should be updated.

• • Classifier error analysis: What do low RB reals look like, what do high RB bogus look like?

• Get light curve observations from reals into training vetted

• Pipeline Analysis to automate:
  

• • What do low RB reals look like, what do high RB bogus look like?

• • Find variables (bogus objects that have multiple alert packets, objects >= n_obs, objects with both positive and negative subtractions?)

• Separate Galactic vs. Extra-Galactic classifiers

• 2018-10 combine_labels.py has an override switch, to override the majority vote for examples that have been revetted
• 2018-11 Filter out old sources (from before Feb 5, 2018)
• 2018-11 Get features from Kowalski vs. IPAC? Kowalski packets are NOT a superset of IPAC db feats! An experiment limited to the intersection of Kowalski and IPAC db feats showed classifier performance decreased! But found workaround for DB performance issues (Frank gave me a way to get nid, rcid from a candid)
• 2018-12 Cross validation performance decreasing steadily since t8 due to persistent contamination within the GROWTH marshall feed
• 2019-01 Get the testing framework into a Jupyter notebook

• Separate Galactic vs. Extra-Galactic

• Flag boguses with light curve observations greater than 2.
• Catch variable stars with isdiffpos=True and false in the light curve

• Kowalski / IPAC candidate discrepancies
• Pixel Clump issues on certain x,y positions (see Ashish's email to Umaa on 6/28/18)

• Known boguses: ZTF18aabtvch, ZTF18aaiafnn, ZTF18aaizvmy

Issues

• Why are candids from IPAC DB not found in alerts db?
• Why is cross validation performance decreasing steadily since t8

• Get light curve observations from reals into training [ POSTPONED - no way to get all confirmed spectroscopic reals ]

DONE

• 2018-08 Correlation between nbad and boguses Report Here

• X is training set version. This is the mix of examples that comprises the training set

• Automated Data Contamination [ Charlotte ] * Find contamination using clustering.
• Active Learning to Improve Training Data Selection [ Sara ] * Use active learning to discover potential batches of boguses (and reals, alternatively)
• Other Data Collection Sources (preferably automated)
  • Find variables (bogus objects that have multiple alert packets, objects >= n_obs, objects with both positive and negative subtractions?)

• Improving Quality of GROWTH Marshall Feed
  • ZTF objects provided in the GROWTH marshall are not necessarily spectroscopically-confirmed, they are saved. Is my list spectroscopically-confirmed? Email to Ashot and Mani

Open Issues / Experiments

• Correlation between nbad and boguses

• Transient Marshal: A weekly feed of reals and boguses are emailed. The boguses are listed by candid with a classification. The reals are listed by ztf object id.
• Zooniverse: . The Galactic Plane was prioritized as an area of improvement by the Science Steering Committee (SSC).

Pipeline Development

• Get light curve observations from reals into training
• Get an override label module (to override the majority vote)
• Add cross matches from catalogs as a source (Nadia's from TNS)
• Kill very old objects (from before Feb 5, 2018)
• Implement proper grid search over RF parameter space
• Get the testing framework into a Jupiter notebook
• Get features from Kowalski vs. IPAC? Do same query against Kowalski and see if we recoup candidateIds
• Pipeline Analysis:
  • Labeled Data Test Set * RB score improvements * Score improvement on known false positive, negatives (Ragnhild's List) * KL divergence between training set, test set features (to find major divergence) * Plot feature distributions on reals vs. boguses * Unlabeled Data ( * Score bias per features

Data Collection

• Find variables
  • bogus objects that have multiple alert packets
  • objects >= n_obs
  • objects with both positive and negative subtractions?
• Automate cross matches from relevant catalogs (e.g., TNS)
• Data contamination
  • ZTF objects provided in the GROWTH marshall are not necessarily spectroscopically-confirmed, they are saved.
  • Some in the science programs are classifying stars as "Bogus"

Open Issues

• Correlation between nbad and boguses?
• Discrete features analysis
• Separate classifiers:
  • Galactic vs. Extra-Galactic
• Use alert data for Real-Bogus in real-time (which means access to 150 features and postage stamps)
• Deep Learning

Papers:

• ML Overview (response to Pub Board comments)
• RB Paper

Overview

The Real-Bogus classifier scores sources on a scale of 0 (bogus) to 1 (real). It is currently a Random Forest classifier that is built upon 'features', which are a collection of statistics and outputs of the real-time data pipeline, that are available in real-time. The classifier is trained on a set of labeled data. Labels are provided via two data collection venues: 1) Zooniverse and 2) GROWTH marshall.

The bogus label should be attributed to artifacts of the telescope optics or the data pipelines. The real label should be attributed to astronomical objects.

Version History

The version tag that Real-Bogus uses is tX_fY_cZ where:

• X is training set version. This is the mix of examples that compromises the training set
• Y is feature version. This is the set of features selected for each example.
• Z is classifier software version. This is the version of the software deployed into the data pipeline. This is not to be confused with the software that generates the training data.

Training Data Collection

• Transient Marshal has been our primary data collection avenue. A weekly feed of reals and boguses are emailed. The boguses are listed by candid with a classification. The reals are listed by ztf object id.
• Zooniverse campaigns have been our primary data collection avenue for the Galactic Plane. The Galactic Plane was prioritized as an area of improvement by the Science Steering Committee (SSC).

To Do List

Tests

• Charlotte Ward's experiments
• Tiara Hung's experiments

• Zooniverse campaigns have been our primary data collection avenue for reals and bogus events. Better subtraction by ZOGY (compared to what was in [i]PTF), combined with stricter (higher) SNR cutoff, means we are receiving fewer bogus events.
• Transient Marshal is our other avenue. SWG members mark events as real or bogus and that provides useful input especially for bogus events, providing feedback to improve the RB scores.
• Archive Marshal (Volunteers needed): input from this stream is not active yet.

• Random Forests: we start on the lines of the [i]PTF model, but use the 68 features output by ZOGY subtractions.
• Deep Learning: Start with triplets (Sci, Ref, Diff), and move on to just (Sci and Ref). Use CNNs.

Output

• Each object gets a score between 0 and 1.
• Smaller RB scores indicates objects that are more likely to be bogus
• Higher scores indicate more real objects
• Models being improved regularly

Data Collection

• Zooniverse
• Transient Marshal
• Archive Marshal (Volunteers needed)

Methods

• Random Forests
• Deep Learning

-- AshishMahabal - 19 Mar 2018