On Dec 5, 2017, at 2:42 PM, Frank Masci wrote: Minutes (12 / 05 / 2017) Present: Ben, Dave S., Ashish, Tom P., Stefanie, Richard W., Umaa, Angie, Jason, Frank * reference images are now being generated for fields with ‘good’ quality data. See Frank for criteria. * difference-images and events are also starting to be generated; tuning is in progress (Frank); difference-image products are being written to archive; alert packets and streak metadata also generated. Events are being extracted from ~90% of the difference-images. Remaining 10% are bad quality, mostly due to bad flat fielding (or position-dependent gain variations). * reprocessing of some commissioning data for reference image and difference image testing (Russ). * linearity analysis (Frank): no significant change in PSF size as a function of flux. Not really conclusive either way. Separate analysis is underway by Michael Medford. * depth-of-coverage map generation now integrated into production pipeline (Ben and Russ). Products are pushed to ztf-depot for collection by the QA monitoring team. Separate products will be available for MSIP-only observations. * automated clean-up of ztfinbox and checking that raw data was archived (Ben) * Stefanie requesting archive-usage statistics. * Frank wrote a stripped down pipeline to generate primarily SExractor catalogs for the flexure analysis. * Dave S. suggested updating pixel-scale thresholds (relative to priors) for flagging bad astrometry to < -0.2% and > +2%. Also reported good news on products generated from 12/4 and 12/5. * Frank to assist Umaa on accessing candidate features and difference-image metadata for ML training framework. * Jason trending biases and flats and updating bad-pixel masks. * plan to decommission ztfweb for archive access on Tuesday January 2nd. * in queue (carried from agenda): - difference-image pipeline tuning; - alert-naming mechanism; - flatfield / LED-config optimization (lo/hi spatial frequency versions); - source-matching/LC pipeline with access to entire matchfiles for galactic science WG. - point source linearity analysis and correction (TBD): stripe 82 analysis, exptime experiment. - ghost mapping and masking; - training RB on diff-image events (involving JPL); - training streak-classifier (solar system WG). - star/galaxy classification scores for PS1 to associate with event stream (A. Millier); - configuring new cluster nodes for production; - archive GUI and associated tools: MOST; finderchart service; lightcurve GUI. Regards, Frank On Nov 28, 2017, at 5:52 PM, Frank Masci wrote: Minutes (11 / 28 / 2017) Present: Dave S., Stefanie, Tom P., Jason, Angie, Ben, Thomas K., Frank PTF: * reprocessing of epochal science frames from second priority list of galactic-plane fields are all done. Next step is to archive the new data following vetting by iPTF team. Frank to discuss schedule with project. After this, we will regenerate reference images for all fields, chips, and filters where data was reprocessed. ZTF: * offline testing/tuning of reference image pipeline for a few fields observed on 11/21 (Frank). Will exercise in production very soon (Russ). * recommendation by Dave S. to update pixel-scale tolerances (relative to priors) for flagging bad frames. New range: +/- 1.5% ? * depth-of-coverage map generation currently being integrated into production pipeline (Ben and Russ). Products will be pushed to ztf-depot for collection by the QA monitoring team. Separate products will be available for MSIP-only observations. * bright source halo masking done (Frank) * conversion of reports for Minor Planet Center to ADES format (in asteroid pipelines) done (Frank) * proposed alert-naming infrastructure design in place -- pending discussion with Steve Groom on mechanism and DB resources. * brief discussion on linearity experiment with good suggestions made by Jason. Not trivial to characterize. Frank will check for flux-dependent PSFs using fwhm and noisepixel estimates. * more documentation updates (Frank) * plan to decommission ztfweb for archive access on Tuesday January 2nd. * In queue (repeated from agenda): - difference-image pipeline tuning; - flatfield / LED-config optimization (lo/hi spatial frequency versions); - point source linearity analysis and correction (TBD); - ghost mapping and masking; - training RB on diff-image events (involving JPL); - star/galaxy classification scores for PS1 to associate with event stream (A. Millier); - configuring new cluster nodes for production; - archive GUI and associated tools: MOST; finderchart service; lightcurve GUI. Regards, Frank On Nov 21, 2017, at 9:28 AM, Frank Masci wrote: Hi All, - Today’s 1PM meeting is cancelled. Here’s a progress report: * PTF galactic plane reprocessing: 488/490 nights done(!) * Assisting with special “serpentine” raster observations to explore image quality vs Dec and hour angle (Dave S) * Data system was prepped to support 19 new fields from Rich D’s boustrophendonic experiment (Russ). Processing of this data is in progress; stay tuned. * Preparing for flat-field experiments and offline processing for validation (Frank) * Made magnitude-limit computations in pipeline more robust (Frank), based on PSF-catalogs; communicated with Michael Medford. * Converted metadata format for ML-vetted streaks from image-differencing pipeline to the new ADES format to support MPC deliveries (Frank) * Archive-access using API tools now ready (Stefanie and IRSA team); email to be sent next week warning users when ztfweb portal will be decommissioned. * Fixed, re-ingested and re-arched raw image files that were delivered with incorrect fieldID length in their names (Russ). This would have messed up archive queries later. Prevention measures implemented to not ingest incorrectly formatted files in future. * Development in progress: - implementing a halo-masking pipeline module with inputs provided by Ginny (Frank) - converting MPC deliverables from ZMODE pipeline to the new ADES format (Frank) - assigning/reusing names for alert packets (Russ and Frank) - configuring and testing new set of 34 cluster nodes (Ben) - coverage-map generator almost ready for production (Ben) - GUI to access archive products (IRSA) On Nov 14, 2017, at 5:43 PM, Frank Masci wrote: Minutes (11 / 14 / 2017) Present: Dave S., Stefanie, Tom P., Tom B., Jason, Angie, Frank * PTF reprocessing: more disk space now available to resume reprocessing; 457 nights out of 490 total in the second priority list are done. ZTF: * Dave S: continuing to assess and tune astrometric calibration. Also, analyzing as a function of airmass. Consistently finding ~ 45 milli-arcsec accuracy (median radial separation w.r.t. Gaia) at an air masses of <~ 2. Request: may want to check performance for each readout-channel separately. Analysis of commissioning data with Anna Ho also in progress. * Related to previous bullet: observations are currently being scheduled above an airmass of 2 (up to 10?). Question for the project: will the survey be limited to a specific airmass cutoff? We vaguely recall 2 being the cutoff, corresponding to ~ 30 deg above the horizon. * Frank: continuing to tune PSF-estimator in pipeline to adapt to elliptical PSFs seen on some chips (primarily corner CCDs 1 and 13); documentation updates; code to convert product formats from asteroid-detection pipeline for submission to MPC to the new ADES format (streaks and tracklets); characterizing instrumental zero-points and magnitude limits from PSF-fit photometry. * Russ: infrastructure to assign names to ZTF alerts prior to delivery to UW in progress; some reprocessing to address failures in the calibration pipelines. * Request to update nightly summary reports with a count of the specific pipeline failures (TBD). * Ghost and bright source halo masking: Frank suggesting that ghost analysis (and mapping) can be done more easily using difference-images (when ready). This helps avoid static-source contaminations. Ghost maps can then be used upstream for masking in science image products. Bright source halos are easier to characterize at present. * Stefanie (and IRSA team): preparing to transition to new API-driven tool to access archive products. * Question raised on when to start creating reference images and tuning that pipeline: Frank -- waiting for some components to stabilize: OS (automated focus/tilt corrections? Flatfield optimization) and DS (further instrumental calibration tuning). We’re very close. Stay tuned (literally). On Nov 7, 2017, at 5:28 PM, Frank Masci wrote: Minutes (11 / 07 / 2017) Present: Dave S., Ben, Stefanie, Thomas K., Tom B., Jason, Ginny, Sarah, Angie, Frank * PTF reprocessing: disks are all full. Will need to borrow more storage to finish off reprocessing with 46 nights to go. ZTF: * Dave S: tuning/monitoring of astrometric calibration and analyses. Found significant dependence in derived pixel scale with airmass (up to 1% deviation). Everything is appropriately caught in astrometric solution. Significant improvements were made over the past week. More tuning will be done (in particular on the distortion prior) following more commissioning experiments. Also updated metric thresholds to catch and report bad astrometry cases for QA. Dave S. also submitted a queue file to support astrometry commission experiment. * Frank: tuning of PSF-estimator in DAOPhot to adapt to elliptical PSFs seen persistently on some chips; offline reprocessing and mosaicking to support next week’s press release; tuning of photometric cal using PS1; explored that phot. zero-points behave as expected with airmass; need more data to assess photometric precision (repeatability); documentation updates; continued feedback/reports to OS/engineering commission team. * Russ: update to nightly summary reports to include breakdown of field / filter / exposure count for current night that was processed. * TBD: request to update nightly summary reports with a count of the specific pipeline failures. * preparing to transition to new API-driven tool to access archive products. Training and testing in progress by Stefanie and Frank. * brief discussion on flat-fielding: prime analyses identified (separate working group): tune illuminator setup and effective wavelengths; shutter timing; temporal stability of overall responsivity; contamination level and location of dust features. * other development in progress: infrastructure to assign names to ZTF alerts prior to delivery to UW; converting formats for submissions to Minor Planet Center (for streaks and tracklets) to the new ADES format (Astrometry Data Exchange Standard); set-up of DB metadata for serving reference image products. On Oct 24, 2017, at 2:52 PM, Frank Masci wrote: Minutes (10 / 24 / 2017) Present: Dave S., Russ, Eric, Tom P., Stefanie, Thomas K., Jason, Matthew, Frank * PTF reprocessing: 401 nights into second (lower-priority) list, with 490 nights in total. ZTF: * zeroth-order WCS parameter file redelivered to camera team earlier today (Dave S). Goal is to integrated this into camera software before we restart observations Thursday night. * Due to tweak in WCS parameters, static Gaia and PS1 calibrator catalogs will be regenerated (Frank and Russ) * analysis of bias and flat data, and stability thereof; conclusion is that flat data is relatively stable (Frank). More flats will be acquired following installation of trim-plate. * pipeline tuning: astrometric calibration (Dave S.); photometric calibration and PSF generation (Frank) * modifying pipeline infrastructure to allow reprocessing and re-archiving at the single exposure level (Russ) * first round of bad-pixel masks to initiate masking in pipeline ready (Jason) * Frank to coordinate with Rich on selected fields to re-observe following trim-plate installation. This is to facilitate a comparison with/without the trim plate. Data will be deposited into its own date-tagged directory in archive so no need to relocate old data. * a few of us (Matthew, Eric, Mansi, Russ, Ben) met earlier today to discuss naming/ID'ing of alerts. Implementation plan is better defined. Some details TBD. * request by Quan-Zhi (Solar System working group and who was not at the meeting) to generate cutouts for known SSOs detected by ZMODE pipeline. This was not in the plan… but discussions are underway. From: Frank Masci Subject: Re: iPTF/ZTF Meeting Today Date: October 17, 2017 at 5:43:23 PM PDT To: Russ Laher , David Shupe , Ben Rusholme , Frank Masci , Carrie Nugent , "Rebbapragada, Umaa D (398J)" , Adam Miller , "Mansi M. Kasliwal" , Eric Bellm , Tom Barlow , Vandana Desai , Jason Surace , David Cook , Thomas A Prince , Yi Mei , Richard Dekany , Lin Yan , Scott Adams , "Kupfer, Thomas" , Kevin Burdge , Matthew Graham , Ulrich Feindt , Jakob Nordin , Deep Chatterjee , prbrady@uwm.edu, Stefanie Wachter Cc: Steve Groom , Imel David Minutes (10 / 17 / 2017) Present: Russ, Jason, Dave S., Tom P., Eric, Stefanie, Thomas K., Frank * PTF reprocessing: 366 nights into second (lower-priority) list, with 490 nights in total. ZTF: * zeroth-order WCS parameter file redelivered to camera team earlier today (Dave S) * PS1 calibrator catalogs now being regenerated using new WCS config (Frank) * Gaia catalogs for astrometry will be generated very soon (Russ) * ad-hoc analysis of bias and flat data, stability thereof (Frank) * read noise and saturation limit estimates for pipeline (Frank) * preparing to make static bad-pixel masks to initiate masking in pipeline (Jason) * Dave S to forward example matchfile from sim data to Kevin B. Minutes (10 / 03 / 2017) Present: Russ, Jason, Ben, Dave S., Deep, Tom P., Eric, Frank * PTF reprocessing: 305 nights into second (lower-priority) list, with 490 nights in total. * new ZTF ops database now ready to support ingestion of real-camera data, including mock/test data (Russ). * continuing to resolve ipac-ingest / camera-software interface issues (Russ). * software to automatically generate and email end-of-night ingest / processing reports (Russ). * mock (source-less) camera data was successfully pushed through the realtime pipeline, with intermediate products in archive and depot; pipelines fail gracefully with expected metrics and error codes. * documenting pipeline-operator checklist and orchestration of daily maintenance tasks (Frank and Russ). * while Eric is in town, Frank to convene meeting on alert naming mechanism and Kafka mirroring infrastructure * Tom P. alerted us that ZTF will need to deliver asteroid ephemeris information (for tracks and streaks) to the MPC in the new ADES format. This format accommodates error-covariance estimates on positions. Useful for orbit determination. Frank to interface with Solar System working group. From: Frank Masci Subject: Re: iPTF/ZTF Meeting Today Date: September 26, 2017 at 2:29:16 PM PDT To: Russ Laher , David Shupe , Ben Rusholme , Frank Masci , Carrie Nugent , "Rebbapragada, Umaa D (398J)" , Adam Miller , "Mansi M. Kasliwal" , Eric Bellm , Tom Barlow , Vandana Desai , Jason Surace , David Cook , Thomas A Prince , Yi Mei , Richard Dekany , Lin Yan , Scott Adams , "Kupfer, Thomas" , Kevin Burdge , Matthew Graham , Ulrich Feindt , Jakob Nordin , Deep Chatterjee , prbrady@uwm.edu Cc: Steve Groom , Imel David Minutes (09 / 26 / 2017) Present: Russ, Ulrich, Jason, Ben, Dave S., Deep, Patrick Brady, Kevin, Frank * PTF reprocessing: 276 nights into second (lower-priority) list, with 490 nights in total. * new ZTF ops database now ready to support ingestion of real-camera data, including mock/test data. Old DB (containing records from simulated data) was copied to a separate server (Russ and Ed) * regenerated static photometric calibrator catalogs from PS1 for new science and engineering field grids; updated color and mag thresholds for calibrator selection. Documentation with new selection criteria and plots also updated (Frank) * updated realtime pipeline to copy alert packets to depot for temporary access by UW; this will be turned off when Kafka mirroring is in place. (Russ) * continuing to ingest and examine camera-like image data; next step is to push this through the realtime pipeline to explore what end ups in archive and depot. This is known to fail because there are no sources in the data (Russ and Frank). * stress-testing of source-matching (lightcurve) pipeline using large batch of simulated catalog data (Dave S.). * presentation on fake-transient methodology for PTF by Deep Chatterjee: https://docs.google.com/presentation/d/1IuqhzrynWYwlrAJORo8EJw4p4_-lbgBOJtTzyjy8VgA Minutes prepended to: http://www.oir.caltech.edu/twiki_ptf/bin/view/ZTF/PipelineMeetingMinutes Regards, Frank On Sep 26, 2017, at 8:18 AM, Frank Masci wrote: Today 1PM, KS-300. Agenda: 1. iPTF - bulk reprocessing update. 2. ZTF pipeline development / testing - routine ingestion / processing of mock camera data - gearing towards processing mock camera data through science pipeline - source-matching pipeline updates - regeneration of static Gaia and PS1 catalogs to support astrometric/photomeric cal. - other --- Goto Meeting: 1. Please join my meeting. https://global.gotomeeting.com/join/595527485 2. Use your microphone and speakers (VoIP) - a headset is recommended. Or, call in using your telephone. Dial +1 (213) 493-0622 Access Code: 595-527-485 Audio PIN: Shown after joining the meeting Meeting ID: 595-527-485 From: Frank Masci Subject: Re: No iPTF/ZTF Meeting Today Date: September 19, 2017 at 2:09:53 PM PDT To: Russ Laher , David Shupe , Ben Rusholme , Frank Masci , Carrie Nugent , "Rebbapragada, Umaa D (398J)" , Adam Miller , "Mansi M. Kasliwal" , Eric Bellm , Tom Barlow , Vandana Desai , Jason Surace , David Cook , Thomas A Prince , Yi Mei , Richard Dekany , Lin Yan , Scott Adams , "Kupfer, Thomas" , Kevin Burdge , Matthew Graham , Ulrich Feindt , Jakob Nordin , Deep Chatterjee Cc: Steve Groom , Imel David Brief progress update: * PTF reprocessing: 252 nights into second (lower-priority) list with 490 nights in total. * regenerating static calibrator catalogs from Gaia and PS1 using new science and engineering field grids; also prepping DB tables using new observing grids. * updated realtime pipeline to copy alert packets to depot for temporary access by UW; this will be turned off when Kafka mirroring is in place. * submitted three new experiment test plans for engineering commissioning (astrometry, biases and dome flats). * continuing to ingest and examine camera-like image data; still working out some missing metadata issues with camera team. * discovered that “seeing” (fwhm) estimate will not be written to raw CCD image headers by camera team. Realtime pipeline was updated to compute this metric on a per-quadrant basis for propagation to product headers. * testing and optimizing source-matching (lightcurve) pipeline using all catalogs generated from simulated data until now. * archive GUI development in progress Regards, Frank From: Frank Masci Subject: Re: iPTF/ZTF Meeting Today Date: September 12, 2017 at 2:13:38 PM PDT To: Russ Laher , David Shupe , Ben Rusholme , Frank Masci , Carrie Nugent , "Rebbapragada, Umaa D (398J)" , Adam Miller , "Mansi M. Kasliwal" , Eric Bellm , Tom Barlow , Vandana Desai , Jason Surace , David Cook , Thomas A Prince , Yi Mei , Richard Dekany , Lin Yan , Scott Adams , "Kupfer, Thomas" , Kevin Burdge , Matthew Graham , Ulrich Feindt , Jakob Nordin Cc: Steve Groom , Imel David Minutes (09 / 12 / 2017) Present: Russ, Ulrich, Jason, Ben, Dave S., Frank ---- iPTF related: * reprocessing of galactic-plane nights: 221 nights into second (lower-priority) list. There are 490 nights in total; these are those with >= 30 exposures at |b| <= 20. ---- ZTF development and testing: * preparing experiment plans for engineering commission; primarily calibration inputs for pipeline: bias, flats, prior distortion (Frank) * initial list of users identified for “ztfweb” access: to facilitate file-system only views to archive and depot (Steve and Frank). Select users were notified with passwords. * continuing to ingest and examine camera-like image data; streaming VPO (virtual PL operator) logic to process different image types (Russ) * sanity checking of raw image data received from camera: ensure identifiers in filenames are consistent with header keywords (Russ) * integration of ZMODE pipeline with VPO (Russ) * archive set-up for reference images with metadata (Russ and Serge) * DB schema design to support object searches for matchfile-based lightcurve retrieval (David S. & Serge) * request: exercise source-matching pipeline on simulated data (Dave and Russ) * chasing zombies on new cluster nodes (Ben) * software for routine coverage-map generation in progress (Ben) A copy of the above is appended to: http://www.oir.caltech.edu/twiki_ptf/bin/view/ZTF/PipelineMeetingMinutes Regards, Frank On Sep 12, 2017, at 9:13 AM, Frank Masci wrote: Today 1PM, KS-300. Agenda: 1. iPTF - bulk reprocessing update. 2. ZTF pipeline development / testing - routine ingestion / processing of mock camera data - [real] archive / depot access - other --- Goto Meeting: 1. Please join my meeting. https://global.gotomeeting.com/join/595527485 2. Use your microphone and speakers (VoIP) - a headset is recommended. Or, call in using your telephone. Dial +1 (213) 493-0622 Access Code: 595-527-485 Audio PIN: Shown after joining the meeting Meeting ID: 595-527-485 Regards, Frank From: Frank Masci Subject: Re: iPTF/ZTF Meeting Today Date: September 5, 2017 at 2:59:25 PM PDT To: Russ Laher , David Shupe , Ben Rusholme , Frank Masci , Carrie Nugent , "Rebbapragada, Umaa D (398J)" , Adam Miller , "Mansi M. Kasliwal" , Eric Bellm , Tom Barlow , Vandana Desai , Jason Surace , David Cook , Thomas A Prince , Yi Mei , Richard Dekany , Lin Yan , Scott Adams , "Kupfer, Thomas" , Kevin Burdge , Matthew Graham , Ulrich Feindt , Jakob Nordin Cc: Steve Groom , Imel David Minutes (09 / 05 / 2017) Present: Russ, Ulrich, Deep, Jason, Ben, Frank ---- iPTF related: * reprocessing of galactic-plane nights: 201 nights into second (lower-priority) list. There are 490 nights in total; these are those with >= 30 exposures at |b| <= 20. * reprocessing has paused because we are output of sandbox (pipeline) disk space. As mentioned previously, we will borrow some storage provisioned for the ZTF archive, then resume reprocessing. This storage will be returned to ZTF. ---- ZTF development and testing: * resumed testing archive ingestion process (includes searchable metadata); updates were made to URL where posts are made (Russ) * continuing to ingest and examine camera-like image data; streaming VPO (virtual PL operator) logic to process different image types (Russ) * currently setting up user-authentication for “ztfweb” to facilitate file-system only views to archive and depot products (Frank & Steve) * assimilation of all new science survey and engineering fieldIDs (tiling grids) to prep for engineering commissioning and SV experiments (Frank) * DB schema design to support archive ingestion of reference image products (Russ) * DB schema design to support object searches for matchfile-based lightcurve retrieval (David S.) * further discussion and thoughts on how to streamline fake transient injection following survey operations (Uli, Deep, Frank) * preparation for internal DS operations readiness review (Frank, Ben, Steve) A copy of the above is appended to: http://www.oir.caltech.edu/twiki_ptf/bin/view/ZTF/PipelineMeetingMinutes Regards, Frank On Sep 5, 2017, at 9:31 AM, Frank Masci wrote: Today 1PM, KS-300. Agenda: 1. iPTF - bulk reprocessing update. 2. ZTF pipeline development - routine ingestion / processing of mock camera data - [real] archive / depot access - db schemas to support reference image product retrieval and lightcurve search via matchfiles. --- Goto Meeting: 1. Please join my meeting. https://global.gotomeeting.com/join/595527485 2. Use your microphone and speakers (VoIP) - a headset is recommended. Or, call in using your telephone. Dial +1 (213) 493-0622 Access Code: 595-527-485 Audio PIN: Shown after joining the meeting Meeting ID: 595-527-485 Regards, Frank From: Frank Masci Subject: No iPTF/ZTF Meeting Today Date: August 29, 2017 at 9:44:13 AM PDT To: Russ Laher , David Shupe , Ben Rusholme , Frank Masci , Carrie Nugent , "Rebbapragada, Umaa D (398J)" , Adam Miller , "Mansi M. Kasliwal" , Eric Bellm , Tom Barlow , Vandana Desai , Jason Surace , David Cook , Thomas A Prince , Yi Mei , Richard Dekany , Lin Yan , Scott Adams , "Kupfer, Thomas" , Kevin Burdge , Matthew Graham , Ulrich Feindt , Jakob Nordin Cc: Steve Groom , Imel David Hi All, A few of us are out of town. Today’s meeting is canceled. Brief report: * reprocessing of iPTF galactic-plane nights to resume shortly following installation of more storage. * continued testing of pipeline-to-ZTF archive interfaces * reference image metadata (schema) for archive currently being assembled * lightcurve (matchfile) metadata for objects table to support LC retrieval also in progress * ztfweb server to access ZTF archive and depot file systems now set up; starting the process of adding users (stay tuned) * resumed ingestion of camera data and ironing out interface details. * ghost/halo masking plan in pipeline devised (Frank); ghost characterization during commissioning TBD. * preparation for upcoming DS operations readiness review. Regards, Frank rom: Frank Masci Subject: Re: iPTF/ZTF Meeting Today Date: August 22, 2017 at 3:43:31 PM PDT To: Russ Laher , David Shupe , Ben Rusholme , Frank Masci , Carrie Nugent , "Rebbapragada, Umaa D (398J)" , Adam Miller , "Mansi M. Kasliwal" , Eric Bellm , Tom Barlow , Vandana Desai , Jason Surace , David Cook , Thomas A Prince , Yi Mei , Richard Dekany , Lin Yan , Scott Adams , "Kupfer, Thomas" , Kevin Burdge , Matthew Graham , Ulrich Feindt Cc: Steve Groom , Imel David Minutes (08 / 22 / 2017) Present: Russ, Ulrich, Deep, Adam, Frank ---- iPTF related: * reprocessing of galactic-plane nights: 194 nights into second (lower-priority) list. There are 490 nights in total; these are those with >= 30 exposures at |b| <= 20. * we are out of sandbox space to continue this reprocessing. The safest option (mentioned by Steve G earlier) is to borrow some storage purchased with ZTF funds to finish up the reprocessing. When done, products will be moved to the PTF archive (with prior project approval of course), and then the disk storage will be returned to ZTF. We do not have the personnel to orchestrate archiving of the reprocessed PTF data right now, and even if we did, this would not be safe without the products first being vetted by the iPTF collaboration. ---- ZTF development and testing: * discussion and thoughts on how to streamline fake transient injection following survey operations. Frank described a minimal impact plan (TBD). * Adam summarized status of S/G classification work and a tentative schedule for delivering PS1 S/G scores to IPAC for use in realtime cross-matching. * continued testing of pipeline/archive interface and hand-off to IRSA. Archive DB server timeouts were seen due to exceeding max number of allowed concurrent connections; will need to increase to accommodate processing (Russ and Serge). * continued work on Virtual Pipeline Operator software (VPO; Russ) * brief discussion on assigning names (and/or alertID) to alert packets. Frank to discuss details with Matthew before proposing a design. * schema of alert packets updated to include three more PS1 mags from realtime association (g): r, i , z. New example avro files (and json equivalents) were pushed to GitHub. * updates to realtime pipeline to ensure that the candidate (opsDB) IDs that are assigned to each event are unique. This will accommodate any reprocessing in future. I.e., we need to ensure that any new packets will contain new candidate IDs to not confuse previously assigned candidateIDs. This will allow traceability back to the pipeline (note: this is not the alert ID mentioned above, which still needs to be fleshed out). * set up of “ztfweb” server to allow archive/depot filesystem visibility in the near future; Frank asked Matthew to compile a list of users for early access using password authentication. Stay tuned. * v1 of ZMODE (moving-object pipeline) delivered to ops (Frank), with test products being written to depot. A copy of the above is appended to: http://www.oir.caltech.edu/twiki_ptf/bin/view/ZTF/PipelineMeetingMinutes On Aug 22, 2017, at 9:32 AM, Frank Masci wrote: Today 1PM, KS-300. Agenda: 1. iPTF - bulk reprocessing update. 2. ZTF pipeline development - big-sim load/performance testing update - processing of mock camera data - pipeline updates (avro schema updates) - ZMODE (asteroid) pipeline ready and tested - [real] archive / depot access --- Goto Meeting: 1. Please join my meeting. https://global.gotomeeting.com/join/595527485 2. Use your microphone and speakers (VoIP) - a headset is recommended. Or, call in using your telephone. Dial +1 (213) 493-0622 Access Code: 595-527-485 Audio PIN: Shown after joining the meeting Meeting ID: 595-527-485 From: Frank Masci Subject: Re: iPTF/ZTF Meeting Today Date: August 15, 2017 at 4:06:35 PM PDT To: Russ Laher , David Shupe , Ben Rusholme , Frank Masci , Carrie Nugent , "Rebbapragada, Umaa D (398J)" , Adam Miller , "Mansi M. Kasliwal" , Eric Bellm , Tom Barlow , Vandana Desai , Jason Surace , David Cook , Thomas A Prince , Yi Mei , Richard Dekany , Lin Yan , Scott Adams , "Kupfer, Thomas" , Kevin Burdge , Matthew Graham , Ulrich Feindt Cc: Steve Groom , Imel David Minutes (08 / 15 / 2017) Present: Ben, Russ, Ulrich, Jason, Frank ---- iPTF related: * reprocessing of galactic-plane nights: 158 nights into second (lower-priority) list. There are 490 nights in total; these are those with >= 30 exposures at |b| <= 20. * related to reprocessing, sandbox 13 is about 50% full; Frank to contact Steve for more disk space or consider moving some data to archive (TBD). Will require consultation with project. ---- ZTF development and testing: * successful ingestion of camera-like data (mostly read-noise) and created a “bias” calibration product (Russ). QA products pushed to ztf-depot. * still resolving some interface / metadata formatting issues with camera team (Frank, Russ). * continued work on Virtual Pipeline Operator software (VPO; Russ and Ben) * continued set up of DB tables and IRSA hand-off to store file products and metadata in archive for raw, cal, sci products (Russ and Serge). * testing and configuring of new 34 compute nodes using new build in progress (Ben and Russ). * new ZOGY module (that was delivered by Brad) has been integrated into realtime pipeline (Frank); this version includes the astrometric-error term for the match-filtered (s/n) product and also fixes a number of bugs. * update of “stars" DB table in transients DB to include more PS1 mags still in progress (Russ and Ed). To include in alert packets from the realtime cross-matching (Frank). * work on the moving-object pipeline (ZMODE) continues - Frank A copy of the above is appended to: http://www.oir.caltech.edu/twiki_ptf/bin/view/ZTF/PipelineMeetingMinutes From: Frank Masci Subject: Re: iPTF/ZTF Meeting Today Date: August 8, 2017 at 2:45:33 PM PDT To: Russ Laher , David Shupe , Ben Rusholme , Frank Masci , Carrie Nugent , "Rebbapragada, Umaa D (398J)" , Adam Miller , "Mansi M. Kasliwal" , Eric Bellm , Tom Barlow , Vandana Desai , Jason Surace , David Cook , Thomas A Prince , Yi Mei , Richard Dekany , Lin Yan , Scott Adams , "Kupfer, Thomas" , Kevin Burdge , Matthew Graham , Ulrich Feindt Cc: Steve Groom , Imel David Minutes (08 / 08 / 2017) Present: Dave S., Ben, Russ, Thomas K., Ulrich, Kevin, Tom B., Frank ---- iPTF related: * reprocessing of galactic-plane nights: 132 nights into second (lower-priority) list. There are 490 nights in total; these are those with >= 30 exposures at |b| <= 20. * related to the reprocessing; we have about 12 days left before sand box 13 fills up. Frank to contact Steve/Eugean. * reprocessing request of realtime data from Quan-Zhi pending (TBD) - to support ML comparisons with JPL. ---- ZTF development and testing: * significant optimization/offloading of DB-oriented processing tasks tied to realtime pipeline (e.g., assembling 30d alert histories). Now using more of a file-system based approach to assist with parallelism instead of placing load on DB server (Russ). * Set up of DB tables to store file product metadata on archive side in progress (Serge, Russ et al.). * new 34 compute nodes were configured and now ready for testing/benchmarking (Ben). SLURM queue was also configured. * nightly and cumulative coverage maps in progress (Ben). Will also be made for MSIP-only data. These are a QA deliverable. * unit-testing of source-matching / relative photometry (Dave S.). Goal is to also test on simulated galactic-plane regions. * alerts from “negative” (ref - new) subtractions now being generated. Associated histories will not condition on sign of event (i.e., both + and - included), nor filter. Only programID (Frank and Russ). * new ZOGY module was delivered by Brad; currently being tested by Frank. Will integrate into realtime pipeline soon. * recoding interface to astrometric calibration module so that prior distortion coefficients are read from a config. file (Dave S.). * continuing to receive test data from camera software; iterating with camera team on missing keywords and formats. * thoughts on how to shoehorn fake-event injection into DS, with minimal impact (Frank) * alert naming mechanism design details and scoping work effort (Frank) * update of “stars" DB table in transients DB to include more PS1 mags in progress. These will be included in alert packets from realtime cross-matching. * brief discussion on programID values attached to each epochal image (Tom B): 0=not science (i.e., cal or engineering); 1=public (msip); 2=collab; 3=caltech. * brief discussion with Tom K on getting MOU in place (involving Tom P) for matchfile products since these will contain epochs for all programIDs. * more work on the moving-object pipeline (ZMODE) - Frank Regards, Frank On Aug 8, 2017, at 8:36 AM, Frank Masci wrote: Today 1PM, KS-300. Agenda: 1. iPTF - bulk reprocessing update. 2. ZTF pipeline development - big-sim load/performance testing update - pipeline updates (new zogy; negative transients; more PS1 mags) - ZMODE (MOPS) pipeline - archive / depot access - ingest test using camera data --- Goto Meeting: 1. Please join my meeting. https://global.gotomeeting.com/join/595527485 2. Use your microphone and speakers (VoIP) - a headset is recommended. Or, call in using your telephone. Dial +1 (213) 493-0622 Access Code: 595-527-485 Audio PIN: Shown after joining the meeting Meeting ID: 595-527-485 From: Frank Masci Subject: Re: iPTF/ZTF Meeting Today Date: July 26, 2017 at 12:12:40 PM PDT To: Russ Laher , David Shupe , Ben Rusholme , Frank Masci , Carrie Nugent , "Rebbapragada, Umaa D (398J)" , Adam Miller , "Mansi M. Kasliwal" , Eric Bellm , Tom Barlow , Vandana Desai , Jason Surace , David Cook , Thomas A Prince , Yi Mei , Richard Dekany , Lin Yan , Scott Adams , "Kupfer, Thomas" , Kevin Burdge , Matthew Graham , Ulrich Feindt Cc: Steve Groom , Imel David Minutes (07 / 25 / 2017) Present: Dave S., Russ, Thomas K., Ulrich, Jason, Frank ---- iPTF related: * reprocessing of galactic-plane nights: 77 nights into second (lower-priority) list. There are 490 nights in total; these are those with >= 30 exposures at |b| <= 20. ---- ZTF development and testing: * yet another 17 nights worth of simulated image data were received (Jason); this set also contains 80 random transients per CCD (20/quad); goal is to stress the system even further (particularly database and alert generation). * implemented process to archive image and catalog files in near-realtime with user metadata (following JSON file spec); this is for raw image files, epoch-based science products, and calibration products (Russ). * new 34 compute nodes are almost ready to roll (tires need to be kicked first). * optimized computation of position-dependent HJD to include in source matchfiles (lightcurve files) - Dave S. * new test camera data were received from camera team; need to ingest and check metadata (Russ and Ben) * preparation for next week’s project meeting. * more work on the moving-object pipeline (ZMODE) - Frank * first draft of pipeline-calibration plan and requirements - Frank Regards, Frank On Jul 25, 2017, at 9:19 AM, Frank Masci wrote: Today 1PM, KS-300. Agenda: 1. iPTF - bulk reprocessing update. 2. ZTF pipeline development - big-sim load/performance testing update - ops / archive interfaces and access - ingest test using camera data - calibration plan --- Goto Meeting: 1. Please join my meeting. https://global.gotomeeting.com/join/595527485 2. Use your microphone and speakers (VoIP) - a headset is recommended. Or, call in using your telephone. Dial +1 (213) 493-0622 Access Code: 595-527-485 Audio PIN: Shown after joining the meeting Meeting ID: 595-527-485 From: Frank Masci Subject: Re: iPTF/ZTF Meeting Today Date: July 18, 2017 at 2:44:24 PM PDT To: Russ Laher , David Shupe , Ben Rusholme , Frank Masci , Carrie Nugent , "Rebbapragada, Umaa D (398J)" , Adam Miller , "Mansi M. Kasliwal" , Eric Bellm , Tom Barlow , Vandana Desai , Jason Surace , David Cook , Thomas A Prince , Yi Mei , Richard Dekany , Lin Yan , Scott Adams , "Kupfer, Thomas" , Kevin Burdge , Matthew Graham , Ulrich Feindt Cc: Steve Groom , Imel David Minutes (07 / 18 / 2017) Present: Dave S., Russ, Thomas K., Ben, Ulrich, Jason, Frank ---- iPTF related: * reprocessing of galactic-plane nights: 52 nights into second (lower-priority) list. There are 490 nights in total; these are those with >= 30 exposures at |b| <= 20. * given the continued (heavy) use of the forced-photometry service, software was updated to be more efficient in terms of disk i/o; products are now generated using the pipeline sandbox instead of the local node (scratch) disks. This avoids clashes with the ongoing reprocessing. ---- ZTF development and testing: * an additional 17 nights worth of simulated image data were received (Jason); this new set contains 80 random transients per CCD (20/quad); goal is to load more transients into the DB for stress testing; processing continues (Russ). * updates made to realtime pipeline to compute some history metrics per transient more efficiently (Russ); primarily ndethist, ncovhist, and associated jdstart, jdend. This uses a filesystem-based approach spread across the compute nodes rather than the single DB server. * realtime pipeline now internally generating v0 alert packets in avro format (Russ). Packet schema still needs a few more fields: primarily to enable archived image traceability and “object” naming (Frank to update) * interface to copy image-calibration products to archive in progress (Russ) * an additional 34 compute nodes were purchased; currently being configured (Ben and IPAC ISG). * decision made to write only mjd and hjd to matchfiles (lightcurve pytables) -- Thomas & Dave S. Since hjd is position dependent, Dave also mentioned that hjd can vary by ~ several seconds on scales of ~ 0.4 deg (typically half a readout-channel). Dave to explore and verify further. * Dave S. will update astrometry code to read distortion prior from a config file (ticket already exists). * further data-link testing (Ben). To convene meeting with Jamie and Patrick. * Frank started to identify and build all pieces needed by the new moving-object pipeline (ZMODE). This pipeline will create moving-object tracks by linking ML-vetted events from the DB and will run at the end of each night where on-sky data was acquired. * IPAC met with camera/OS team yesterday to discuss near-term schedule and commissioning (contact Frank for minutes). Regards, Frank On Jul 18, 2017, at 8:23 AM, Frank Masci wrote: Hi All, Today 1PM, KS-300. Agenda: 1. iPTF - bulk reprocessing update. 2. ZTF pipeline development - big-sim load/performance testing update - coverage map generation - ops / archive interfaces and access - documentation - calibration plan --- Goto Meeting: 1. Please join my meeting. https://global.gotomeeting.com/join/595527485 2. Use your microphone and speakers (VoIP) - a headset is recommended. Or, call in using your telephone. Dial +1 (213) 493-0622 Access Code: 595-527-485 Audio PIN: Shown after joining the meeting Meeting ID: 595-527-485 Regards, Frank From: Frank Masci Subject: Re: iPTF/ZTF Meeting Today Date: June 13, 2017 at 3:45:51 PM PDT To: Russ Laher , David Shupe , Ben Rusholme , Frank Masci , Carrie Nugent , "Rebbapragada, Umaa D (398J)" , Adam Miller , "Mansi M. Kasliwal" , Eric Bellm , Tom Barlow , Vandana Desai , Jason Surace , David Cook , Thomas A Prince , Yi Mei , Richard Dekany , Lin Yan , Scott Adams , "Kupfer, Thomas" , Kevin Burdge , Matthew Graham , Ulrich Feindt Cc: Steve Groom , Imel David Minutes (06 / 13 / 2017) Present: Dave S., Ulrich, Jason, Russ, Mansi, Ben, Thomas K., Kevin, Frank ---- iPTF related: * all 165 nights from first list of high-priority list are done. * we are currently 6 nights into the second (lower-priority) list of 490 nights; these are those nights with >= 30 exposures at |b| <= 20. * Thomas K. reported a significant number of epochs with missing difference-image extractions from reprocessing of high-cadence galactic plane fields from Jul 2016; some cases traced to bad input reference images. ---- ZTF development and testing: * testing on 34-night galactic-plane simulation set (Russ): 24 nights processed; code optimization continues, particularly on DB architecture to retain as much history as possible (30 day max per event at this time). * schema on alert-packet generation: request from Mansi to assign a unique name to each candidate extracted by pipeline and all positionally-associated historical events included in a packet. This name needs to follow IAU conventions (e.g., ZTF[18abc]…?); Frank: presumably, this name need to back-propagate into the pipeline DB so that future (new) candidates can inherit the same name as any pre-named associated historical events, even if it’s junk… right?; very messy in confused/high-density fields(!); also messy when we reprocess portions of the data (e.g. due to cal/astrometry/ref-image refinements); lots of architecture/DB/design details involved, recalling that only T-30days worth of history will be available during *realtime processing*, i.e., for efficient generation and delivery of alerts to UW. Frank to discuss details and resources with project scientist. * obsHJD now computed and inserted into matchfile products; filename mnemonic that encodes date/time-stamp is also now propagated to facilitate image retrieval (Dave S.). New ICD and example matchfile available for perusal. * discussion with camera-team on required metadata for raw-image headers (Frank, Reed. Roger…) * Further updates to documentation. http://web.ipac.caltech.edu/staff/fmasci/home/junk/ztf-archive/ztf_pipelines_deliverables_v1.0.pdf *** I will be out of the office from June 15 to July 10 but will be in email contact. Response times will be slower than usual. Please contact Ben Rusholme for urgent matters. We will not have this meeting for the next three weeks. Progress updates will still be emailed. Regards, Frank On Jun 13, 2017, at 9:28 AM, Frank Masci wrote: Hi All, Today 1PM, KS-300. Agenda: 1. iPTF - bulk reprocessing update: all 165 nights from first list of high-priority list done. - Now working on second list of 490 lower-priority nights -- defined as those nights with >= 30 exposures at |b| <= 20. 2. ZTF pipeline development - big-sim load/performance testing updates and code restructuring - alert distribution: first version of schemas ready for review; details TBD. - coverage map generation - source-matching / lightcurve generation updates - ops / archive interfaces and access - documentation --- Goto Meeting: 1. Please join my meeting. https://global.gotomeeting.com/join/595527485 2. Use your microphone and speakers (VoIP) - a headset is recommended. Or, call in using your telephone. Dial +1 (213) 493-0622 Access Code: 595-527-485 Audio PIN: Shown after joining the meeting Meeting ID: 595-527-485 From: Frank Masci Subject: Re: iPTF/ZTF Meeting Today Date: June 6, 2017 at 4:24:07 PM PDT To: Russ Laher , David Shupe , Ben Rusholme , Frank Masci , Carrie Nugent , "Rebbapragada, Umaa D (398J)" , Adam Miller , "Mansi M. Kasliwal" , Eric Bellm , Tom Barlow , Vandana Desai , Jason Surace , David Cook , Thomas A Prince , Yi Mei , Richard Dekany , Lin Yan , Scott Adams , "Kupfer, Thomas" , Kevin Burdge , Matthew Graham , Ulrich Feindt Cc: Steve Groom , Imel David Minutes (06 / 06 / 2017) Present: Dave S., Ulrich, Jason, Russ, Adam, Tom B., Tom P., Thomas K., Kevin, Frank ---- iPTF related: * ~ 148 nights out of 165 from high priority list have been reprocessed. A second list of 490 nights with >= 30 exposures at |b| <= 20 has been prepped. Started to run into disk space issues. Will provision another 64 TB to move forward. Will also consider removing sandbox data associated with unwanted fields. ---- ZTF development and testing: * Discussed new sky-grid layouts (Tom B.) and importance of having the primary and secondary (overlapping) grids defined early on. These grids cover ~ 99% of the sky. Distribution of remaining 1% to be explored. Also somewhat related: only declinations >~ -30 deg matter for ZTF, or rather, the PS1 boundary. * Related to previous bullet: whether proposed tiling scheme can accommodate future ubercal by collaboration. Understanding is that enough field-to-field overlap and continuity across sky is needed to enable this. Uli agreed to check with the Stockholm team, Jakob Nordin et al. * Request for obsHJD in matchfile products (Dave S. to check). Also, a new ICD with example matchfiles are ready for perusal. Dave S. to send to Thomas K. and Kevin. * Testing on 34-night galactic-plane simulation set (Russ): 14 nights processed; optimization continues, particularly on retaining as much history as possible per event. * Alert-packet generation prototyping in progress (Frank). New schemas defined following pipeline updates. * Documentation: small updates (Frank). Primarily to photometric calibration section (http://web.ipac.caltech.edu/staff/fmasci/home/junk/ztf-archive/ztf_pipelines_deliverables_v1.0.pdf) * Raw-camera image metadata interface specs discussed with camera team; required in order to ingest/process any on-sky data at IPAC in future, including engineering data. * Carried over from a previous meeting: Frank worked with Umaa to define initial interface for point-source RealBogus in pipeline. Regards, Frank On Jun 6, 2017, at 8:54 AM, Frank Masci wrote: Hi All, Today 1PM, KS-300. Agenda: 1. iPTF - bulk reprocessing update: 148 nights out of 165 from high priority list done; second list of 490 nights with >= 30 exposures at |b| <= 20 ready to go. 2. ZTF pipeline development - big-sim testing update - alert distribution: first version of schemas and alert-packet (avro) generation code ready. - coverage map generation - ops / archive interfaces and access - raw image metadata interface with camera software (reminder for Rich) --- Goto Meeting: 1. Please join my meeting. https://global.gotomeeting.com/join/595527485 2. Use your microphone and speakers (VoIP) - a headset is recommended. Or, call in using your telephone. Dial +1 (213) 493-0622 Access Code: 595-527-485 Audio PIN: Shown after joining the meeting Meeting ID: 595-527-485 From: Frank Masci Subject: Re: iPTF/ZTF Meeting Today Date: May 23, 2017 at 4:23:52 PM PDT To: Russ Laher , David Shupe , Ben Rusholme , Frank Masci , Carrie Nugent , "Rebbapragada, Umaa D (398J)" , Adam Miller , "Mansi M. Kasliwal" , Eric Bellm , Tom Barlow , Vandana Desai , Jason Surace , David Cook , Thomas A Prince , Yi Mei , Richard Dekany , Lin Yan , Scott Adams , "Kupfer, Thomas" , Kevin Burdge , Matthew Graham , Ulrich Feindt Cc: Steve Groom , Imel David Minutes (05 / 23 / 2017) Present: Dave S., Ulrich, Ben, Jason, Tom B., Frank ---- iPTF related: * ~ 104 nights from the “high-priority” list of 165 galactic-plane nights have been reprocessed. After this, we will revisit original list of 1400 nights and prioritize further if needed. * data from high-cadence galactic-plane experiment (Jul 2016) reprocessed through image-differencing and delivered. ---- ZTF development and testing: * Throughput testing on 34-night galactic-plane simulation set continues; code updated in response to findings. * Frank explored new cuts on PS1 catalog (derived from MeanObject table) to support photometric calibration. Readout-channel-based photometric-calibration tables are being regenerated using new cuts (primarily mitigation of flux-variable sources). * Ben has implemented v0 of coverage-map generation software. Plan is to generate both an end-of-night and cumulative (past survey) coverage-map for each filter, in Aitoff-Hammer projection. * Brief discussion with Uli on incorporating fake-transient injection in pipeline; Frank suggests having a separate data stream that is not part of the production pipeline or archive, nor released to the community. This separate stream will consist of a reprocessed subset of the on-sky data but with the inclusion of fake-transients modeled from lightcurves. Schedule TBD. * Tom B. has defined new pointing grids for ZTF (starting with Eran’s initial layout); team will be asked for feedback soon. * Frank and Steve catching up on latest presentations on alert distribution infrastructure. * Frank worked with Umaa to define initial interface for point-source RealBogus in pipeline. Regards, Frank On May 23, 2017, at 8:54 AM, Frank Masci wrote: Hi All, Today 1PM, KS-300. Agenda: 1. iPTF - bulk reprocessing update (104 of 165 high-priority nights complete) 2. ZTF pipeline development - big-sim testing update - PS1 catalog from MeanObject table: ingestion, filtering and analysis for photometric cal. - coverage map generation - ops / archive interfaces --- Goto Meeting: 1. Please join my meeting. https://global.gotomeeting.com/join/595527485 2. Use your microphone and speakers (VoIP) - a headset is recommended. Or, call in using your telephone. Dial +1 (213) 493-0622 Access Code: 595-527-485 Audio PIN: Shown after joining the meeting Meeting ID: 595-527-485 From: Frank Masci Subject: Re: iPTF/ZTF Meeting Today Date: May 9, 2017 at 3:20:29 PM PDT To: Russ Laher , David Shupe , Ben Rusholme , Frank Masci , Carrie Nugent , "Rebbapragada, Umaa D (398J)" , Adam Miller , "Mansi M. Kasliwal" , Eric Bellm , Tom Barlow , Vandana Desai , Jason Surace , David Cook , Thomas A Prince , Yi Mei , Richard Dekany , Lin Yan , Scott Adams , "Kupfer, Thomas" , Kevin Burdge , Matthew Graham , Ulrich Feindt Cc: Steve Groom , Imel David Minutes (05 / 09 / 2017) Present: Russ, Mansi, Ben, Thomas K., Kevin, Jason, Dave C., Ulrich, Frank ---- iPTF related: * ~ 67 nights from the “high-priority” list of 165 galactic-plane nights have been reprocessed. * reference images for the ~30 high cadence fields from July 2016 were made (Russ). Now running image-subtraction pipeline. Frank to forward Thomas K examples. * Frank and Russ helped out with missing data issues for Halpha and products from g + i experiment. Dave C. will work with Frank to resolve Ha issues. Goal is to reprocess ~237 Ha chips (at least). ---- ZTF development and testing: * Russ reported that reference image generation from 34 night sim-set are almost done. These will be used in image-subtraction pipeline and DB stress testing very soon. * Frank updated ref-image pipeline to fix a small bug related to rotation of the output footprint as derived from input image orientations. * Ben: data link performance still holding up; from last week: median transfer time per exposure was ~20.1 sec; 99% of exposures arrived within ~33 sec, -- comfortably within the 45 sec acquisition interval. Two exposures arrived at >45 sec. * Ben: continuing work on survey-coverage map generator; both incremental (end of night) and cumulative (during survey). * Frank downloaded new PS1 catalog (origin is MeanObject table at MAST); Frank to filter and prepare for loading into the ops DB to serve two purposes: (i) generation of filtered table files to support realtime photometric calibration; (ii) associate with realtime event stream with S/G flags. The latter has involved discussions with A. Miller. * brief discussion on fake-transient injection pipeline; pending further discussion with project scientist and others. * alert distribution strategy (point-source component of ztf-depot): details still under discussion. * pipeline-to-archive interface design, and archive filesystem structure in progress. * some participants requested list of available technical projects. Here it is: http://www.oir.caltech.edu/twiki_ptf/bin/view/ZTF/AvailableTechnicalProjects Regards, Frank On May 9, 2017, at 8:46 AM, Frank Masci wrote: Hi All, Today 1PM, KS-300. Agenda: 1. iPTF - bulk reprocessing update; - realtime rerun for galactic-plane (high cadence) fields from Jul 2016; - missing reference images for g+i expmt since Nov 2016. - missing archive data for Ha (237 chips needed for paper) 2. ZTF pipeline development - big-sim testing update - new PS1 catalog from MeanObject table now in flat files at IPAC - ops to archive interfaces - ZTF-depot / alert-distribution interface pending --- Goto Meeting: 1. Please join my meeting. https://global.gotomeeting.com/join/595527485 2. Use your microphone and speakers (VoIP) - a headset is recommended. Or, call in using your telephone. Dial +1 (213) 493-0622 Access Code: 595-527-485 Audio PIN: Shown after joining the meeting Meeting ID: 595-527-485 Regards, Frank From: Frank Masci Subject: Re: iPTF/ZTF Meeting Today Date: May 2, 2017 at 3:50:41 PM PDT To: Russ Laher , David Shupe , Ben Rusholme , Frank Masci , Carrie Nugent , "Rebbapragada, Umaa D (398J)" , Adam Miller , "Mansi M. Kasliwal" , Eric Bellm , Tom Barlow , Vandana Desai , Jason Surace , David Cook , Thomas A Prince , Yi Mei , Richard Dekany , Lin Yan , Scott Adams , "Kupfer, Thomas" , Kevin Burdge , Matthew Graham Cc: Steve Groom , Imel David Minutes (04 / 25 / 2017) Present: Dave S., Tom P., Russ, Ben, Thomas K., Tom B., Adam, Jason, Frank ---- iPTF related: * 57 nights from the “high-priority” list of 165 galactic-plane nights have been reprocessed. * plans are underway to generate reference images for the ~30 high cadence fields from July 2016. * Frank to help out with missing data issues for Halpha and products from g + i experiment. ---- ZTF development and testing: * ingesting and executing instrumental-calibration pipeline on new 34-night simulation set. 27 nights ingested; proceeding with instrumental calibration so can generate reference images to see image subtraction later on (Russ) * source-matching / relative photometry: tweaks to clean up unnecessary file paths; command-line switches to always compute lightcurve metrics - Dave S. To forward a new example to Thomas K for marshal testing. * Tom P. asked about schedule for matchfile access through archive for marshal testing; Frank - tentatively within the next several weeks; simulated versions can be made available sooner. Another issue is that if marshal is ingesting entire matchfiles, they will need to apply proper masking of proprietary epochs therein (caltech vs msip vs partner time) prior to distribution. * Tom B. mentioned including additional sequential counters in headers of raw images from scheduling. * Ben implemented throughput / data-link simulation monitor. Recent test shows that median transfer time per exposure was ~20.1 sec; 99% of exposures arrived within ~33 sec, -- comfortably within the 45 sec acquisition interval. Two exposures arrived at >45 sec. * Adam M distributed some code to perform more reliable S/G classification on PS1 “StackObjects” table. Plan is to ingest S/G scores into IPAC’s “stars” table at a later date. Frank to send Adam example schema of IPAC version. * ZTF pipeline s/w is now on Github. Send Frank your github username if interested. * alert distribution strategy (subset of ztf-depot): details under discussion. Stay tuned. Regards, Frank On May 2, 2017, at 8:45 AM, Frank Masci wrote: Hi All, Today 1PM, KS-300. Agenda: 1. iPTF - reprocessing update; plans for high-cadence fields: ref-image building to kick off soon. 2. ZTF pipeline development - big-sim testing update - ZTF-depot / alert-distribution status - Transition to Git/GitHub done - provisioning ML (RB) for point source events [reminder for others] --- Goto Meeting: 1. Please join my meeting. https://global.gotomeeting.com/join/595527485 2. Use your microphone and speakers (VoIP) - a headset is recommended. Or, call in using your telephone. Dial +1 (213) 493-0622 Access Code: 595-527-485 Audio PIN: Shown after joining the meeting Meeting ID: 595-527-485 From: Frank Masci Subject: Re: iPTF/ZTF Meeting Today Date: April 25, 2017 at 3:09:46 PM PDT To: Russ Laher , David Shupe , Ben Rusholme , Frank Masci , Carrie Nugent , "Rebbapragada, Umaa D (398J)" , Adam Miller , "Mansi M. Kasliwal" , Eric Bellm , Tom Barlow , Vandana Desai , Jason Surace , David Cook , Thomas A Prince , Yi Mei , Richard Dekany , Lin Yan , Scott Adams , "Kupfer, Thomas" , Kevin Burdge , Matthew Graham Cc: Steve Groom , Imel David Minutes (04 / 25 / 2017) Present: Dave S., Russ, Jason, Ben, Frank ---- iPTF related: * 34 nights from the “high-priority” list of 165 nights have been reprocessed. That’s ~12 nights / calendar week, bearing in mind these contain the galactic plane and will take longer to process. Pipeline was re-engineered to use all 24 cluster machines instead of 11 (per chip model) - Russ. * Waiting on Thomas K. to provide feedback on science-frame products reprocessed from the Jul’16 high cadence experiment. ---- ZTF development and testing: * started ingesting and executing instrumental-calibration pipeline on new 34-night simulation set. Goal is to stress system (filesystems and databases). * current testing uses a new version of the VPO (Virtual Pipeline Operator) - Russ. * loading of “forced-phot" PS1 into ops DB with S/G flags to support event cross-matching - Done by Ed Jackson. Ready for testing in realtime pipeline. * compute dynamic fixed-aperture corrections using PSF-fit and aperture instrumental photometry in realtime pipeline - Frank; * implemented a "broken trail” detector for realtime pipeline, i.e., to catch blinking/glinting aircraft and satellites - Frank. Sources associated with these trails are masked and omitted from downstream consumption. * transition to Git/GitHub for ZTF pipeline s/w will happen at 2PM sharp on April 26, 2017. * source-matching / relative photometry : tweaks being made to clean up unnecessary file paths; command-line switches to always compute lightcurve metrics, irrespective of relative photometry - Dave S. * alert distribution strategy (subset of ztf-depot): still being discussed at Cahill. Regards, Frank On Apr 25, 2017, at 8:43 AM, Frank Masci wrote: Hi All, Today 1PM, KS-300. Agenda: 1. iPTF - reprocessing update; plans for high-cadence fields - start ref-image building. Waiting to feedback. 2. ZTF pipeline development - prep to run big simulation set - realtime pipeline updates - loading of “forced-phot" PS1 into ops DB with S/G flags to support cross-matching - ZTF-depot / alert-distribution status - Transition to Git/GitHub - provisioning ML (RB) for point source events [reminder for others] --- Goto Meeting: 1. Please join my meeting. https://global.gotomeeting.com/join/595527485 2. Use your microphone and speakers (VoIP) - a headset is recommended. Or, call in using your telephone. Dial +1 (213) 493-0622 Access Code: 595-527-485 Audio PIN: Shown after joining the meeting Meeting ID: 595-527-485 Minutes (04 / 18 / 2017) Present: Dave S., Russ, Tom P., Thomas K., Kevin, Jason, Tom B., Matthew, Frank ---- iPTF related: * 50 nights have been reprocessed through the science-frame pipeline; Frank to send Thomas K. paths to products from Jul’16 high-cadence galactic plane experiment. Following vetting, plan is to proceed with ref-image generation for these only. ---- ZTF development and testing: * 32 nights worth of simulated images are done; waiting for another 2 nights and a summary file (Jason). Expect to start processing this set early next week. * Virtual Pipeline Operator (VPO) software in progress (Russ). This does not replace a human pipeline operator. * New streak functionality (streak-PSF-fitter) and ML-classifier is now in realtime pipeline (Frank). Annotated cutouts are also generated on ML-vetted streak candidates (Frank). * Related to the above: new schema for “streaks” DB table is now in place (Russ). * In progress: loading of (forced photometry based) PS1 catalog into operations DB with s/g classification flags to support real-time event association (Russ). Will be replaced with the “MeanObject” version later and s/g flags will be replaced with probability-based scores. * Matchfile optimization (Dave S.): pre-sorting on source-ID speeds up relative photometry step and LC retrieval downstream (I think that’s what I heard). * Plan to transition to GitHub (ZTF development account) on April 24. Frank mentioned whether ZTF project-based github account can be linked (perhaps for specific users at first) to new account. Ben to check with David I. Russ mentioned some proprietary code needs to be omitted from GitHub (e.g., DAOPhot modules, other?). * Request by Galactic Marshal (Thomas K. and Tom P.): ability to download entire matchfiles from IRSA when new versions are available. Question arose on how to mask-out epochs acquired under Caltech time before delivery. If Galactic Marshal is the only customer, then perhaps marshal could mask these prior to use(?) TBD. * Brief ZTF-Depot discussion: can be broken up into three major themes: (i) point source events; (ii) solar system products (mops and streaks); (iii) QA. There are plans to only serve (i) through an alert-distribution system - details TBD. (ii) and (iii) will be accessed by their respective marshals (one or two people in each case). * Provisioning a machine-learned classifier (RealBogus) for point sources for execution in the realtime pipeline. Frank alerted Tom P. that it’s best to start planning and notify the JPL team. Even though algorithms are much improved, Frank thinks we need a safety net. We are being fooled by the near-perfect simulated data and not enough testing on real (PTF) data. * Brief discussion on survey tiling grids. IPAC requests that the grids be defined ~ 3 weeks prior to start of science-verification so that various calibration catalogs can be pre-generated (for more efficient retrieval under the parallel filesystem). Regards, Frank On Apr 18, 2017, at 8:44 AM, Frank Masci wrote: Hi All, Today 1PM, KS-300. Agenda: 1. iPTF - reprocessing update; plans for high-cadence fields - start ref-image building soon. 2. ZTF pipeline development - big simulation update (multi-night with galactic plane) - realtime pipeline updates (streaks, history metric redesign,..) - loading of “forced-phot" PS1 into ops DB with S/G flags to support cross-matching - ZTF-depot / alert-distribution status - provisioning ML (RB) for point source events --- Goto Meeting: 1. Please join my meeting. https://global.gotomeeting.com/join/595527485 2. Use your microphone and speakers (VoIP) - a headset is recommended. Or, call in using your telephone. Dial +1 (213) 493-0622 Access Code: 595-527-485 Audio PIN: Shown after joining the meeting Meeting ID: 595-527-485 From: Frank Masci Subject: Re: iPTF/ZTF Meeting Today Date: April 11, 2017 at 2:41:51 PM PDT To: Russ Laher , David Shupe , Ben Rusholme , Frank Masci , Carrie Nugent , "Rebbapragada, Umaa D (398J)" , Adam Miller , "Mansi M. Kasliwal" , Eric Bellm , Tom Barlow , Vandana Desai , Jason Surace , David Cook , Thomas A Prince , Yi Mei , Richard Dekany , Lin Yan , Scott Adams , "Kupfer, Thomas" , Kevin Burdge , Matthew Graham Cc: Steve Groom , Imel David Minutes (04 / 11 / 2017) Present: Russ, Tom P., Eric, Adam M., Thomas K., Ben, Jason, Frank ---- iPTF related: * reprocessing update: proceeding with galactic plane processing (|b| <~ 22deg). Thomas K. to provide prioritized list of nights containing fields of interest. Plan is to immediately create ref images and rerun realtime PL on these high priority fields once science-frames are ready. * Thomas K. also to review galactic plane fields with very few exposures and whether corresponding nights should be reprocessed. ---- ZTF development and testing: * multi-night images simulations are in progress (by Jason); 12 days possible(?). Plan is to replicate this and fudge observation times to make a bigger dataset. * Frank to ensure more disk space added to /ztf/simdata/ (16TB was requested). * Ben waiting for access to a P48 machine. Optimistic that access will be granted this week. * Ben almost ready to transition to GitHub (ZTF development account). Frank mentioned whether ZTF project-based github account can be linked (perhaps for specific users at first) to new account. Ben to check with David I. Russ mentioned some proprietary code needs to be omitted from GitHub (e.g., DAOPhot modules, other?). * First transfer of raw camera data from Camera Team into IPAC inbox occurred late last week. Transfer successful. Packaging of data needs a little more work. * More throughput testing and adjustments to realtime pipeline (Russ). Frank to propose a new design for some of the more “expensive” historical-based candidate metrics. Stay tuned. * Frank, Steve G., Eric, Matthew G., Shri, Mansi and others had a productive meeting at UW last week. Topic was the Kafka alert-distribution system. Frank, Steve, Matthew started to strategize on alternatives to ZTF-Depot, at least for distributing alerts. * Discussion on obtaining a new version of PS1 (more accurate version using MeanObjects table instead). Frank requesting Adam to send new filters used to perform simple S/G classification. Frank then to apply cuts on IPAC's PS1 catalog to facilitate realtime cross-matching. * Frank started to import and interface new streak-vetting code into realtime pipeline. Also spending considerable time on tuning MODE software (Moving Object Discovery Engine). On Apr 11, 2017, at 8:48 AM, Frank Masci wrote: Hi All, Today 1PM, KS-300. Agenda: 1. iPTF - reprocessing update: TBD: which nights to prioritize/omit? 2. ZTF pipeline development - big simulation update (multi-night with galactic plane) - realtime pipeline updates (image subtraction and more metrics) - prep for streaking asteroid functionality - prep for new PS1 catalog: S/G flags and transient candidate cross-matching. - ZTF-depot status - latest on throughput testing - misc. --- Goto Meeting: 1. Please join my meeting. https://global.gotomeeting.com/join/595527485 2. Use your microphone and speakers (VoIP) - a headset is recommended. Or, call in using your telephone. Dial +1 (213) 493-0622 Access Code: 595-527-485 Audio PIN: Shown after joining the meeting Meeting ID: 595-527-485 Regards, Frank From: Frank Masci Subject: Re: iPTF/ZTF Meeting Today Date: March 28, 2017 at 8:59:29 PM PDT To: Russ Laher , David Shupe , Ben Rusholme , Frank Masci , Carrie Nugent , "Rebbapragada, Umaa D (398J)" , Adam Miller , "Mansi M. Kasliwal" , Eric Bellm , Tom Barlow , Vandana Desai , Jason Surace , David Cook , Thomas A Prince , Yi Mei , Richard Dekany , Lin Yan , Scott Adams , "Kupfer, Thomas" , Kevin Burdge Cc: Steve Groom , Imel David Minutes (03 / 28 / 2017) Present: Dave S., Russ, Tom P., Thomas K., Ben, Adam, Jason, Frank ---- iPTF related: * reprocessing status: M31 (6/2012 - 2/2017) image-differencing done (Frank to forward location to Thomas). * frameproc for galactic plane (|b| < 22d) nights: Russ to update p13 pipeline for efficiency (to ensure new astrometry and pv2sip code is used). Will need to progressively archive due to disk space reasons; * Frank to check with Wei to ensure any reprocessed data from now on is not visible by public. ---- ZTF development and testing: * Ben waiting for access to a P48 machine (to contact R. Walters when returns). Also met with Reed to chat about transferring camera-softare generated data. * source-matching/relative photometry: now in production (Russ); Dave experimenting with matchfile compression. Also found that sorting by matchID in matchfiles improves speed of lightcurve retrieval by up to 40x. Final delivered matchfiles (to IRSA and perhaps gal-marshal) will most likely be sorted. * four additional history-related metrics are now computed per candidate and pushed to depot (Russ); “history” implies going back to beginning of survey; metrics are: (i) the number of historical candidates that are spatially coincident with a given candidate within X arcsec; (ii) the number of historical difference images that touch its position; (iii) and (iv) the jdstart and jdend spanned by the earlierst and latest detections respectively. * new database table created to store information for scheduled fields from two sky grids: center pointings,and corners of all 64 readout-channels per field. This is provisional and it’s understood that the sky tiling is not yet defined. IPAC requests that once finalized, these baseline grids be used for the entire survey, however, one is free to schedule new field IDs if needed. * Frank to check with Eric on status of schedule file so Jason can proceed with new sims: primarily ~30 continuous nights crossing the galactic plane. * Frank working on implementing aperture corrections for the SExtractor (aperture) photometry catalogs; to ensure consistency with accompanying PSF-fit catalogs. * Frank working on ZMODE: the ZTF Moving Object Discovery Engine; inherited from PTFMOPS; also looking to sell this as “LMODE”, where L=LSST. On Mar 28, 2017, at 9:41 AM, Frank Masci wrote: Hi All, Today 1PM, KS-300. Agenda: 1. iPTF - reprocessing status (M31 - p15; gal-plane - p5; starflat nights - p5). 2. ZTF pipeline development - resumption of data transfer testing - request for a multi-night simulation that includes galactic plane: use internal “scheduling simulator”. - integration of source matching / relative photometry - transition to Git / GitHub - misc. --- Goto Meeting: 1. Please join my meeting. https://global.gotomeeting.com/join/595527485 2. Use your microphone and speakers (VoIP) - a headset is recommended. Or, call in using your telephone. Dial +1 (213) 493-0622 Access Code: 595-527-485 Audio PIN: Shown after joining the meeting Meeting ID: 595-527-485 Regards, Frank Regards, Frank Begin forwarded message: From: Frank Masci Subject: Re: iPTF/ZTF Meeting Today Date: March 14, 2017 at 3:03:13 PM PDT To: Russ Laher , David Shupe , Ben Rusholme , Frank Masci , Carrie Nugent , "Rebbapragada, Umaa D (398J)" , Adam Miller , "Mansi M. Kasliwal" , Eric Bellm , Tom Barlow , Vandana Desai , Jason Surace , David Cook , Thomas A Prince , Yi Mei , Richard Dekany , Lin Yan , Scott Adams , "Kupfer, Thomas" , Kevin Burdge Cc: Steve Groom , Imel David Minutes (03 / 14 / 2017) Present: Russ, Adam, Ben, Jason, Tom B., Dave S., Frank ---- iPTF related: * new 2-pass SCAMP code and PV2SIP conversion code was integrated and uni-tested in frameproc pipeline to support reprocessing. A single night (July 3, 2016) containing galactic plane observations will be reprocessed later today. This will be ready for analysis early tomorrow. * "On-the-fly” forced photometry requests were not working for i-band. Fixed by Russ. All requests were resubmitted. ---- ZTF development and testing: * link/transfer testing: radio dish now fixed; could not resume throughput testing because a PTF computer at P48 was brought down (Ben). R. Walters notified. * source-matching/relative photometry: integration into production environment in progress. To do: (i) attach/propagate program ids to epochs; (ii) rename output files to standard ztf format, e.g: ztf___c_q_srcmatch.hdf(?) and ztf___c_q_srcmatchlog.txt(?); (iii) ensure switches are in place to only perform source-matching and metric computations, or everything: source-matching + metric-computations + relative photometry. * more experimentation with ZOGY image-subtraction on PTF data using M51; implemented relative-gain correction module (Frank); new results were posted. * discussion on simulating galactic plane images with no input schedule file… Frank needs to check with Eric. Follow-on from previous meeting: * Mansi to provide IPAC with a revised Local-Universe (LU) table to support cross-matching (drawn from NED); will replace entries in existing version. Regards, Frank On Mar 14, 2017, at 8:21 AM, Frank Masci wrote: Hi All, Today 1PM, KS-300. Agenda: 1. iPTF - code updates to frameproc for reprocessing. unit test on galactic-plane night: 2016-07-03; cf. to previous. 2. ZTF pipeline development - resumption of data transfer testing - request for a multi-night simulation that includes a large swath of galactic plane [pending] - integration of source matching / relative photometry - ZOGY testing on PTF data (M51 revisited) - transition to Git / GitHub - coverage-map generation software --- Goto Meeting: 1. Please join my meeting. https://global.gotomeeting.com/join/595527485 2. Use your microphone and speakers (VoIP) - a headset is recommended. Or, call in using your telephone. Dial +1 (213) 493-0622 Access Code: 595-527-485 Audio PIN: Shown after joining the meeting Meeting ID: 595-527-485 From: Frank Masci Subject: Re: iPTF/ZTF Meeting Today Date: March 7, 2017 at 4:03:22 PM PST To: Russ Laher , David Shupe , Ben Rusholme , Frank Masci , Carrie Nugent , "Rebbapragada, Umaa D (398J)" , Adam Miller , "Mansi M. Kasliwal" , Eric Bellm , Tom Barlow , Vandana Desai , Jason Surace , David Cook , Thomas A Prince , Yi Mei , Richard Dekany , Lin Yan , Scott Adams , "Kupfer, Thomas" , Kevin Burdge Cc: Steve Groom , Imel David Minutes (03 / 07 / 2017) Present: Russ, Mansi, Adam, Ben, Jason, Frank ---- iPTF related: * new 2-pass SCAMP code and PV2SIP conversion code to be integrated and tested in frameproc pipeline to support reprocessing. Will reprocess a single field/chip to check. Goal: code integration this week; unit test early next week. * reprocessed star-flat experiment data using refined flats from Jakob. ---- ZTF development and testing: * link/transfer testing: throughput went back down; problem was reported with a radio dish in HPWREN. * source-matching/relative photometry ready; to be integrated into pipeline; request to propagate program id information too. * more experimentation with ZOGY image-subtraction algorithm on PTF data using M51; Frank to advertise results on a web server. * request for multi-night simulation set (goal: >= 10 days that cross the galactic plane); Frank to check resources. * ongoing testing and tuning of ztf-depot product generation (Russ and Frank). New example case posted. * Frank had splinter discussion with Mansi and Adam on strategies for external-catalog cross-matching in realtime pipeline. Plan: v1 will involve cross-matching to the PS1 MeanObjects table with a pre-assigned ternary star-classification flag: 0=star; 0.5=unknown; 1=galaxy. v2 (TBD) will replace this flag with a machined-learned probabilistic score, computed using the deeper PS1 StackObjects table. Three [maybe four] columns will suffice for realtime matching: ra, dec, [rmag], score. * Mansi to provide IPAC with a revised Local-Universe (LU) table to support cross-matching (drawn from NED); will replace entries in existing version. * Mansi asked for five additional columns for the candidates table in the realtime operations DB. All sound feasible. Regards, Frank On Mar 7, 2017, at 9:00 AM, Frank Masci wrote: Hi All, Today 1PM, KS-300. Agenda: 1. iPTF - code updates / unit testing for reprocessing. - status of missing ptfide products, e.g., e.g., fieldid 4466, ccd 9, g+i from 2/2/17 - reprocessing of star-flat experiment data - It’s a wrap. PTF close-out. 2. ZTF pipeline development - data transfer/link testing - request for a multi-night simulation that includes a large swath of galactic plane - integration of source matching / relative photometry - ztf-depot product generation and optimization - cross-matching to external catalogs in realtime pipeline (***PS1 only?) - ZOGY experimentation / tuning on PTF data (M51 revisited) - transition to Git / GitHub; testing using new build from fresh checkout - coverage-map generation software --- Goto Meeting: 1. Please join my meeting. https://global.gotomeeting.com/join/595527485 2. Use your microphone and speakers (VoIP) - a headset is recommended. Or, call in using your telephone. Dial +1 (213) 493-0622 Access Code: 595-527-485 Audio PIN: Shown after joining the meeting Meeting ID: 595-527-485 From: Frank Masci Subject: Re: iPTF/ZTF Meeting Today Date: February 28, 2017 at 4:47:17 PM PST To: Russ Laher , David Shupe , Ben Rusholme , Frank Masci , Carrie Nugent , "Rebbapragada, Umaa D (398J)" , Adam Miller , "Mansi M. Kasliwal" , Eric Bellm , Tom Barlow , Vandana Desai , Jason Surace , David Cook , Thomas A Prince , Yi Mei , Richard Dekany , Lin Yan , Scott Adams , "Kupfer, Thomas" , Kevin Burdge Cc: Steve Groom , Imel David Minutes (02 / 28 / 2017) Present: Dave S., Mansi, Russ, Tom B., Adam, Ben, Frank ---- iPTF related: * discussion on preparation for reprocessing. Dave S. to experiment with 2-pass SCAMP code (as run in realtime pipeline) on a science image with retention of SIP distortion coefficients. Will need to modify PV-to-SIP code to ensure headers can continue to have both distortion representations for archive. * Dec encoder problems were encountered. Images may have pointing offsets of up to a few arc minutes from nominal. Unsure if IPAC pipeline can still calibrate these. * We have another two nights of operations left, weather permitting. R-band only. Robotic scheduling disabled. Will need to modify raw images with more metadata for ingestion at IPAC (Russ) * Scott identified that a significant number of images were not processed by PTFIDE. e.g., fieldid 4466, ccd 9, g+i from 2/2/17. Russ to retrace processing steps and products. * in queue: reprocessing of "star-flat experiment" nights using new star-flats from Jakob; goal: understand spatial variations in system throughput (Frank and Ron) ---- ZTF development and testing: * near-term schedule discussion, primarily reinitiating discussions with Camera team on (i) obtaining lab data to confirm specs (before ~March 10); (ii) sketch out data-link/transfer protocols (before ~ March 20); (iii) have data-transfer software in place to transfer simulated data from summit and explore compression (~ end of April). * source-matching/relative photometry code delivered (Dave S.); interface specs for pipeline integration forwarded to Russ. ICD also written (Dave S.). * more experimentation with ZOGY image-subtraction algorithm on PTF data, using M51 (Frank). Number of false positives is ~6-15 per PTF chip. More testing needed. * new image simulations seeded using PS1 (Jason) successfully processed (Russ). New simulation is for a Winter Solstice night with more point source transients than previous simulations. Can process a 12hr night in ~ 8.6hr. * ongoing testing and tuning of ztf-depot product generation (Russ and Frank). Expect to showcase a new nightly run to collaboration soon. * Frank had discussion with NED team on possibility of cross-matching event stream with NED DB and returning metrics of interest. Their schedule for developing required APIs (for use on other projects at least) is mid Summer. Not viable in near term. Frank to communicate with project on fixing the external catalog for basic cross-matching in realtime pipeline. Regards, Frank On Feb 28, 2017, at 9:08 AM, Frank Masci wrote: Hi All, Today 1PM, KS-300. Agenda: 1. iPTF - preparation for reprocessing (see plan from last week). - other high priority fields to reprocess besides galactic plane and M31? 2. ZTF pipeline development status - near-term schedule and reminders - source matching/relative photometry update - ztf-depot product generation and optimization - transition to Git / GitHub; testing using new build from fresh checkout - coverage-map generation software. - other --- Goto Meeting: 1. Please join my meeting. https://global.gotomeeting.com/join/595527485 2. Use your microphone and speakers (VoIP) - a headset is recommended. Or, call in using your telephone. Dial +1 (213) 493-0622 Access Code: 595-527-485 Audio PIN: Shown after joining the meeting Meeting ID: 595-527-485 From: Frank Masci Subject: Re: iPTF/ZTF Meeting Today Date: February 21, 2017 at 5:06:04 PM PST To: Russ Laher , David Shupe , Ben Rusholme , Frank Masci , Carrie Nugent , "Rebbapragada, Umaa D (398J)" , Adam Miller , "Mansi M. Kasliwal" , Eric Bellm , Tom Barlow , Vandana Desai , Jason Surace , David Cook , Thomas A Prince , Yi Mei , Richard Dekany , Lin Yan , Scott Adams , "Kupfer, Thomas" , Kevin Burdge Cc: Steve Groom , Imel David Minutes (02 / 21 / 2017) Present: Dave S., Jason, Thomas K., Kevin, Frank ---- iPTF related: * new classifier for streaking asteroids now in iPTF production (Frank & Brian from JPL) * new/revamped streak-PSF fitting module also deployed to iPTF production (Frank); to compare with ML classifier. * reprocessed "star-flat experiment" nights using new star-flats from Jakob; goal: understand spatial variations in system throughput. * brief discussion on reprocessing plan; pipeline refinements to commence in early March (see initial agenda below for plan). ---- ZTF development and testing: * offline discussion: received feedback on photometric calibration using PS1; need to replace PS1 catalog with “MeanObject” table that includes cross-epoch StdDev estimates to weed out variables, amongst other cuts (Frank); Eric to ask Brani for snapshot. Currently, the ForcedMeanObject table is used in pipeline -- shows significantly more scatter than the MeanObject photometry. * discussion on analysis comparing SDSS photometry with PS1 (Kevin). Report is available from Kevin. * discussion on definition of photometric zeropoint under “basic calibration scheme”, i.e., need to correct derived ZP to a fixed spectral template, which is not necessarily the zero color (intercept) solution from fit; details TBD (Jason). * new image simulations seeded using PS1 (Jason) successfully ingested (Russ). Subset reprocessed through a number of pipeline subsystems (Russ). * ongoing pipeline stress testing; monitoring of CPU utilization, filesystems, and DB servers (all). * analysis and continued streamlining / parallelizing of ztf-depot product generation (Russ and Frank). Regards, Frank On Feb 21, 2017, at 9:14 AM, Frank Masci wrote: Hi All, Today 1PM, KS-300. Agenda: 1. iPTF - new classifier for streaking asteroids now in production - new streak fitter in production. - reprocessing plan overview/steps: (i) update astrometric cal. in frameproc; check other dependencies downstream; understand archive inconsistencies. (ii) reprocess a night of gal-plane data through frameproc (e.g., high-cadence fields from Jul’16); (iii) recreate reference images (iv) check quality of (ii) and (iii), cf. to previous products (v) rerun image-differencing with new candidates into transients DB (vi) check quality of (v) to decide if worth proceeding with image-differencing for all fields below (vii) reprocess all fields at |b| < 20d and all M31 going back to 1/1/13. Ensure IRSA DB-load files made for sources table. (viii) create new references from (vii) (ix) contingent on quality from (vi), rerun image-differencing on all outputs from (vii) (x) run source-matching on all reprocessed fields going back to 3/1/2009 (xi) regenerate IRSA DB-load files from new matchfiles to support LC retrieval for DR4 2. ZTF pipeline development status - recent pipeline tweaks/updates - source matching/relative photometry update - ztf-depot product generation and optimization - transition to Git / GitHub; testing using new build from fresh checkout - coverage-map generation software. - other --- Goto Meeting: 1. Please join my meeting. https://global.gotomeeting.com/join/595527485 2. Use your microphone and speakers (VoIP) - a headset is recommended. Or, call in using your telephone. Dial +1 (213) 493-0622 Access Code: 595-527-485 Audio PIN: Shown after joining the meeting Meeting ID: 595-527-485 From: Frank Masci Subject: Re: iPTF/ZTF Meeting Today Date: February 14, 2017 at 5:02:28 PM PST To: Russ Laher , David Shupe , Ben Rusholme , Frank Masci , Carrie Nugent , "Rebbapragada, Umaa D (398J)" , Adam Miller , "Mansi M. Kasliwal" , Eric Bellm , Tom Barlow , Vandana Desai , Jason Surace , David Cook , Thomas A Prince , Yi Mei , Richard Dekany , Lin Yan , Scott Adams , "Kupfer, Thomas" Cc: Steve Groom , Imel David Minutes (02 / 14 / 2017) Present: Russ, Ben, Jason, Thomas K., Frank ---- iPTF related: * integration of new classifier for streaking asteroids in progress; goal: tonight. * reprocessed "star-flat experiment" nights using new star-flats from Jakob; goal: better understand spatial variations in throughput for ZTF. * brief discussion on reprocessing plan: at least high-cadence observations of gal-plane from July 2016, M31 going back to 1/1/2013, including realtime pipeline; Frank to ask team for additional fields of interest. ---- ZTF development and testing: * provisional PS1 (DR1) catalog cuts made to support photometric calibration; table files now ready for pipeline use (Frank). * new simulated image set delivered: uses PS1 as seed catalog with no mag limit; includes more transients and better bookkeeping of location (Jason). * new image simulations successfully ingested (Russ). * Frank to convene meeting with Thomas K. and Kevin to describe matchfile-generation / lightcurve-retrieval model. * pipeline stress testing; monitoring of CPU utilization, filesystems, and DB servers in progress (Russ, Ben, Dave F., Steve ...). Ben to update SLURM config. to enable 32 threads / node. * streamlining / parallelizing design of ztf-depot product generation (Russ). Considering generation of "aggregate statistics” file via a cron-job. * discussions with Nadia regarding which external catalogs to use for cross-matching event stream (Frank). * pipeline testing using fresh checkout from Git / GitHub; including RTB (Russ). * coverage-map generation software in progress (Ben); useful for visualizing sim data. On Feb 14, 2017, at 9:08 AM, Frank Masci wrote: Hi All, Today 1PM, KS-300. Agenda: 1. iPTF - new classifier for streaking asteroids ready; to integrate today - reprocessing of star-flat experiment data - reprocessing plan and priorities: photometric PL w/ new astrom; new ref images; realtime PL (galactic-plane, M31, other fields?) - other updates 2. ZTF pipeline development status - PS1 (DR1) catalog cuts and static table files now ready to support photometric calibration. - new image simulation seeded by PS1 (DR1) ready for testing - source matching/relative photometry: serving lightcurves through “object-lookup” interface; need splinter mtg. - ztf-depot product generation and optimization. - transition to Git / GitHub; testing using new build from fresh checkout. - coverage-map generation software. --- Goto Meeting: 1. Please join my meeting. https://global.gotomeeting.com/join/595527485 2. Use your microphone and speakers (VoIP) - a headset is recommended. Or, call in using your telephone. Dial +1 (213) 493-0622 Access Code: 595-527-485 Audio PIN: Shown after joining the meeting Meeting ID: 595-527-485 From: Frank Masci Subject: Re: iPTF/ZTF Meeting Today Date: February 7, 2017 at 2:26:53 PM PST To: Russ Laher , David Shupe , Ben Rusholme , Frank Masci , Carrie Nugent , "Rebbapragada, Umaa D (398J)" , Adam Miller , "Mansi M. Kasliwal" , Eric Bellm , Tom Barlow , Vandana Desai , Jason Surace , David Cook , Thomas A Prince , Yi Mei , Richard Dekany , Lin Yan , Scott Adams , "Kupfer, Thomas" Cc: Steve Groom , Imel David Minutes (02 / 07 / 2017) Present: Dave S., Russ, Ben, Jason, Tom P., Tom B., Frank ---- iPTF related: * i-band references for new fields from Tom B. were generated (Russ). * realtime pipeline was updated to execute new "streak-PSF-fiting” module (Frank). Analysis in progress. * retraining of new classifier for streaks (in progress by Brian at JPL; expect new training model end of this week). ---- ZTF development and testing: * provisional PS1 (DR1) catalog cuts made to support photometric calibration in realtime pipeline; static catalogs now being made in operations environment; see http://www.oir.caltech.edu/twiki_ptf/bin/view/ZTF/PhotometricCalibration * analysis and derivation of S/G flags using PS1 DR1 to assist in associating with event stream from realtime pipeline; in progress (Frank). * Jason updated image simulator to use PS1 as seed catalog. Re-simulated Winter solstice night. Ready for pipeline ingestion/processing end of this week. This run still a paucity of low galactic latitude fields. * source-matching / relative photometry unit testing in progress (Dave). * pipeline stress testing; monitoring of CPU utilization, filesystems, and DB servers in progress (Russ, Ben, Dave F. and Systems Group). Request by Ben to execute 32 instances per machine instead of 8. * streamlining / parallelizing design of ztf-depot product generation (Russ). * transition to Git / GitHub; fresh checkout and software build done [170202]; ready for testing (Ben). Frank suggested setting up a Regression Test Baseline (RTB) to compare against previous run [160830]. * brief discussion with Tom P. on galactic marshal interface; retrieval of object-based lightcurve metrics. Frank to convene meeting with Thomas K. and Kevin to discuss details. Regards, Frank On Feb 7, 2017, at 8:45 AM, Frank Masci wrote: Hi All, Today 1PM, KS-300. Agenda: 1. iPTF - new software for streak identification now in realtime pipeline. - retraining of new classifier for streaking asteroids [in progress]. - other updates 2. ZTF pipeline development status - provisional PS1 (DR1) catalog cuts made; static table files to support photometric calibration now in progress. - derivation of S/G flags from PS1 DR1 to assist in associating with event stream from realtime pipeline [project decision] - image-simulation updates - source matching/relative photometry update. - ztf-depot product generation. - transition to Git / GitHub, testing using new build; visibility to ZTF collaboration. - coverage-map generation software. --- Goto Meeting: 1. Please join my meeting. https://global.gotomeeting.com/join/595527485 2. Use your microphone and speakers (VoIP) - a headset is recommended. Or, call in using your telephone. Dial +1 (213) 493-0622 Access Code: 595-527-485 Audio PIN: Shown after joining the meeting Meeting ID: 595-527-485 From: Frank Masci Subject: Re: iPTF/ZTF Meeting Today Date: January 31, 2017 at 3:17:50 PM PST To: Russ Laher , David Shupe , Ben Rusholme , Frank Masci , Carrie Nugent , "Rebbapragada, Umaa D (398J)" , Adam Miller , "Mansi M. Kasliwal" , Eric Bellm , Tom Barlow , Vandana Desai , Jason Surace , David Cook , Thomas A Prince , Yi Mei , Richard Dekany , Lin Yan , Scott Adams Cc: Steve Groom , Imel David Minutes (01 / 31 / 2017) Present: Dave S., Russ, Ben, Jason, Frank ---- iPTF related: * construction of new g- and i-band references all up to date (Russ). Currently being analyzed by Scott Adams. * realtime pipeline updated to execute streak “PSF-fitter” (Frank). Analysis in progress by Quan-Zhi. * retraining of new classifier for streaks (in progress by JPL team). * discussion of reprocessing plan following decommissioning of PTF observing system. Follow-on from a previous meeting: (i) reprocess science frames using new astrometry then archive (fields TBD); (ii) recreate reference images; (iii) rerun realtime PL on swaths of gal-plane; (iv) rerun source-matching/rel-phot for all of PTF/iPTF. ---- ZTF development and testing: * PS1 DR1 is now in a local database to support analyses (Russ and Ed). Russ examined spatial coverage variations to check for holes. Frank to provide cuts to support photcal. Will then reexamine spatial coverage to identify “uncalibratable” fields/chips. * analysis of PS1 catalog for photometric calibration (Frank; see http://www.oir.caltech.edu/twiki_ptf/bin/view/ZTF/PhotometricCalibration). Received good feedback from Eran O. Discussion on differential airmass corrections. Frank to explore. * analysis and derivation of S/G flags using PS1 DR1 to assist in associating with event stream from realtime pipeline (Frank). * generation of “static” calibrator catalogs to support photometric calibration in pipeline (Frank) * Jason updated image simulator to use PS1 as seed catalog. Also a request to simulate more point source transients. * source-matching / relative photometry unit testing in progress (Dave). * streamlining / parallelizing design of ztf-depot to support realtime discovery (Russ). * transition to Git / GitHub; cleanup underway; goal is to build and test from a fresh checkout before making official (Ben) Regards, Frank On Jan 31, 2017, at 8:42 AM, Frank Masci wrote: Hi All, Today 1PM, KS-300. Agenda: 1. iPTF - g-band reference image generation status. - new software for testing streak identification in realtime pipeline. - retraining new classifier for streaking asteroids. - reprocessing plans, new astrometry code in frameproc, switch to pure TPV format for distortion. 2. ZTF pipeline development status - construction of local PS1 catalog [to support phot. cal and provide S/G metrics] - source matching/relative photometry update. - ztf-depot product generation. - transition to Git / GitHub with visibility to ZTF collaboration. - coverage-map generation software. --- Goto Meeting: 1. Please join my meeting. https://global.gotomeeting.com/join/595527485 2. Use your microphone and speakers (VoIP) - a headset is recommended. Or, call in using your telephone. Dial +1 (213) 493-0622 Access Code: 595-527-485 Audio PIN: Shown after joining the meeting Meeting ID: 595-527-485 From: Frank Masci Subject: Re: iPTF/ZTF Meeting Today Date: January 18, 2017 at 8:34:36 AM PST To: Russ Laher , David Shupe , Ben Rusholme , Frank Masci , Carrie Nugent , "Rebbapragada, Umaa D (398J)" , Adam Miller , "Mansi M. Kasliwal" , Eric Bellm , Tom Barlow , Vandana Desai , Jason Surace , David Cook , Thomas A Prince , Yi Mei , Richard Dekany , Lin Yan , Scott Adams Cc: Steve Groom , Imel David Minutes (01 / 17 / 2017) Present: Dave S., Russ, Ben, Tom B., Thomas K., Jason, Tom P., Frank ---- iPTF related: * status of inaccessible products from archive: all appears to be resolved and in place, including Halpha. * construction of new g-band references in progress: Tom B. sent 148 new fields; only criterion to enforce is >= 5 frames. * possible issue with IMGTYPE keyword in scheduling of science observations for night of Jan 9/10. No problem on scheduling end. To be explored in pipeline operations (database ID: itid). * Mansi requested a dump of the entire PTF Transients Database for use in researching DB technologies for ZTF Marshals (e.g., MyriaDB) … in progress by Ed Jackson. * Discussion of reprocessing plan following decommissioning of PTF observing system. Follow-on from a previous meeting: (i) reprocess science frames using new astrometry then archive (fields TBD); (ii) recreate reference images; (iii) rerun realtime PL on swaths of g-plan; (iv) rerun source-matching/rel-phot for all of PTF/iPTF. ---- ZTF development and testing: * Construction of PS1 catalog for internal use: Jason reformatting a copy obtained from Carl Grillmair; ascii tables for bulk-loading into DB almost ready; max number of rows is ~ 2.7B, 13 columns all floats; can be brought down to ~ 0.6B with a mag cut. Storage and DB location TBD. * source-matching / relative photometry unit testing: runtime and memory now within bounds [Dave S.]. Integrate into production pipeline soon. Need to think about epoch-dependent proprietary tagging, i.e., for exposures acquired under caltech, collaboration, or public (MSIP) time. Ben suggested a time-series “mask”. * generation of products for ztf-depot to support realtime discovery: first version done [Russ & Frank]. Frank notified team of example products: http://web.ipac.caltech.edu/staff/fmasci/home/junk/ztf-depot/ * Frank also notified team of example products for the long-term archive: http://web.ipac.caltech.edu/staff/fmasci/home/junk/ztf-archive/ * transition to Git / GitHub; goal: end of Jan. Need to clean up current CVS. [Ben; in progress] * coverage-map generation software [Ben; in progress] * bandwidth testing from P48 to IPAC: Anna(?) from Cahill tasked by Shri to help analyze variability in HPWREN. Regards, Frank On Jan 17, 2017, at 9:13 AM, Frank Masci wrote: Hi All, Today 1PM, KS-300. Agenda: 1. iPTF - status of unarchived/inaccessible products from archive [Ha night 10/11/16?; some nights with no phot. cal]. - g-band reference image generation for recent i-band fields: 148 field IDs sent by Tom B. - retraining new classifier for streaking asteroids [in progress; delivery delayed due to emergency in JPL team]. 2. ZTF pipeline development status - construction of local PS1 catalog [to support phot. cal and provide S/G metrics] - source matching/relative photometry update. - ztf-depot product generation [mock products to be made visible soon]. - ztf-archive products to be made visible too. - transition to Git / GitHub with visibility to ZTF collaboration. - coverage-map generation software. --- Goto Meeting: 1. Please join my meeting. https://global.gotomeeting.com/join/595527485 2. Use your microphone and speakers (VoIP) - a headset is recommended. Or, call in using your telephone. Dial +1 (213) 493-0622 Access Code: 595-527-485 Audio PIN: Shown after joining the meeting Meeting ID: 595-527-485 From: Frank Masci Subject: Re: iPTF/ZTF Meeting Today Date: January 10, 2017 at 4:54:17 PM PST To: Russ Laher , David Shupe , Ben Rusholme , Frank Masci , Carrie Nugent , "Rebbapragada, Umaa D (398J)" , Adam Miller , "Mansi M. Kasliwal" , Eric Bellm , Tom Barlow , Vandana Desai , Jason Surace , David Cook , Thomas A Prince , Yi Mei , Richard Dekany , Lin Yan , Scott Adams Cc: Steve Groom , Imel David Minutes (01 / 10 / 2017) Present: Dave S., Eric, Scott, Russ, Ben, Tom B., Thomas K., Jason, Frank ---- iPTF related: * status of inaccessible products from archive: apart for some H-alpha nights(?), appears to be settled. Need clarification from Russ and Wei. * i-band reference image generation: ~2000 images recently made for new fields, including g-band. Need to make references for nights of Jan 6/7. * need to relax input criteria for generating references for recently scheduled fields (requested by Mansi); only require at least 5 frame overlaps. Desire to regenerate all references for recent fields using relaxed criteria if possible. Plan is _not_ to archive these for now. * possible issue with IMGTYPE keyword in scheduling of science observations for night of Jan 9/10, Tom B? (this is translated to “itid” in operations DB); used to decide if image is of “science” type for processing by relevant pipelines. * issues with percentile metrics in lightcurve DB products: origin is probably source-matching/rel-phot code; code was identified by Dave S. To be explored. ---- ZTF development and testing: * Experimentation and vetting of PS1 catalog: Carl Grillmair bulk-downloaded a version. Plan to further filter and reformat for ZTF use: primarily (i) photometric calibration; (ii) provide S/G metadata for vetted transient candidates; (iii) seed the image simulator to get representative galactic-plane data for pipeline [Jason and Frank] * source-matching / relative photometry unit testing in progress; memory issue solved! [Dave S.] * generation of products for ztf-depot in progress [Russ and Frank]. Frank to provide example products and layout for team later this week. * updates to instrumental calibration pipeline to mask more flavors of bad pixels [Frank; in progress] * transition to Git / GitHub; goal: end of Jan [Ben; in progress] * coverage-map generation software [Ben; in progress] * bandwidth from P48 to IPAC nominal again, confirmed via iperf test [Ben] Regards, Frank On Jan 10, 2017, at 8:50 AM, Frank Masci wrote: Hi All, Today 1PM, KS-300. Agenda: 1. iPTF - status of unarchived/inaccessible products from archive [still missing Ha night 10/11/16; some nights with no phot. cal. - Rex]. - i-band reference image generation / usage in realtime pipeline; need i-band refs for fields observed on Jan 6/7. - retraining new classifier for streaking asteroids [in progress; delivery delayed due to emergency in JPL team]. - issues with percentile metrics in lightcurve DB [update cautionary notes]. 2. ZTF pipeline development status - construction of local PS1 catalog [to support phot. cal and provide S/G metrics; Frank working with Carl G. and Brani] - source matching/relative photometry update. - ztf-depot product generation [make versions with mock products visible this week]. - schedule for transitioning to Git / GitHub with visibility to ZTF collaboration. - routine coverage-map generation software. - other updates: e.g., updates to pixel masking as per Eran’s suggestions [Frank]. --- Goto Meeting: 1. Please join my meeting. https://global.gotomeeting.com/join/595527485 2. Use your microphone and speakers (VoIP) - a headset is recommended. Or, call in using your telephone. Dial +1 (213) 493-0622 Access Code: 595-527-485 Audio PIN: Shown after joining the meeting Meeting ID: 595-527-485 From: Frank Masci Subject: Re: iPTF/ZTF Meeting Today Date: December 20, 2016 at 5:49:29 PM PST To: Russ Laher , David Shupe , Ben Rusholme , Frank Masci , Carrie Nugent , "Rebbapragada, Umaa D (398J)" , Adam Miller , "Mansi M. Kasliwal" , Eric Bellm , Tom Barlow , Vandana Desai , Jason Surace , David Cook , Thomas A Prince , Yi Mei , Richard Dekany , Lin Yan , Scott Adams Cc: Steve Groom , Dave Flynn , Imel David Minutes (12 / 20 / 2016) Present: Dave S., Eric, Russ, Tom P., Jason, Frank ---- iPTF related: * Inaccessible iPTF archive products: Russ to speak with Wei from IRSA. Includes reprocessed nights and Halpha-only nights; resetting of public-release dates on reprocessed nights that are already public. * Request to only copy required products to ptf-depot from realtime pipeline for remainder of iPTF (Done). * Reprocessing of real streaking asteroids to retrain new ML-classifier in progress; want to assess performance for remainder of iPTF. * iPTF member reported a problem with percentile metrics in lightcurve DB; Eric mentioned that input data used to compute metrics on pipeline side is different than what is presented to user via Source DB. Resolution and documentation in progress. * Dave S: point Frank to results of testing new astrometry (reSCAMP) code on PTF challenge fields. Frank then to find corresponding products in archive to forward to Eric. * At end of iPTF, request to reprocess galactic plane using new astrometry (reSCAMP) code that has been running in realtime PL, but not in photometric PL; then regenerate reference images for use in realtime PL; then to reprocess same data through realtime PL with new references. * At end of iPTF, also request to reprocess all M31 data (a few chips?) through realtime PL. * At end of iPTF, also to run source matching and relative photometry on all (reprocessed) PTF data; regenerate DB load files for IRSA, etc.. etc.. ---- ZTF development and testing: * PS1 public release was on 12/19/16. Several of us experimenting with queries; Brani is going to provide Eric and Carl G. with a LSD version; Carl may also be downloading his own version; IPAC may ingest either of these versions. May also consider seeding image simulator with PS1. Frank to ping Steve on IRSA’s plans (if any). * source-matching / relative photometry unit testing in progress (Dave S.) -- successfully executed on replicated simulated catalog data (~ 80M sources spanning 500 nights). Relative photometry appears to be consuming more memory than expected; profiling is in progress. Stored functions to retrieve inputs were implemented (Russ). * generation of products for ztf-depot in progress: candidate metadata -- Russ; image cutouts -- Frank. * Ben and Frank met with Camera Team to discuss image-packaging and data transfers, with scheduled end-of-calibration and end-of-night notifications in the form of manifest summary files. Also request by project to ingest and archive the guide and focus CCD image data, amounting to ~ 2% more raw data. Transfer of these ancillary images will occur at end of on-sky operations, not in realtime. * Frank experimenting with filters in realtime image-subtraction pipeline to purify event stream; also working with Eran to tune ZOGY algorithm. * Tom P. mentioned possibility of more NSF funding to support anticipated shortfalls in development across project -- primarily MSIP deliverables. This includes hardware. Frank (with Steve) to communicate areas of possible concern. * Transition to Git / GitHub (Ben; in progress). * coverage-map generation software (Ben; in progress). Regards, Frank On Dec 20, 2016, at 8:19 AM, Frank Masci wrote: Hi All, Today 1PM, KS-300. Agenda: 1. iPTF - missing and/or inaccessible products in archive [resolution in progress] - archived difference images now available through ptf-depot (since ~ Oct 2015; to complement scanning) - copying of selected realtime pipeline products to ptf-depot. Not all. - request (from Tom B?): get list of specific fields that will be used for g + i experiments in Jan. - reprocessing of real streaking asteroids in progress: to support training of new classifier 2. ZTF pipeline development status - PS1 catalog release and constructing a “clean” local copy of exclusively point sources to support photometric calibration. - related to previous point: update image simulator to use PS1 as input? - R&D in filtering of extractions from ZOGY outputs (good progress). - source matching/relative photometry update - ztf-depot and archive design in progress: make versions with mock products visible to collaboration very soon. - stamp-image cutouts now being generated by realtime pipeline (a depot deliverable). - schedule for transitioning to Git / GitHub with visibility to ZTF collaboration. --- Goto Meeting: 1. Please join my meeting. https://global.gotomeeting.com/join/595527485 2. Use your microphone and speakers (VoIP) - a headset is recommended. Or, call in using your telephone. Dial +1 (213) 493-0622 Access Code: 595-527-485 Audio PIN: Shown after joining the meeting Meeting ID: 595-527-485 From: Frank Masci Subject: Re: iPTF/ZTF Meeting Today Date: December 13, 2016 at 2:38:30 PM PST To: Russ Laher , David Shupe , Ben Rusholme , Frank Masci , Carrie Nugent , "Rebbapragada, Umaa D (398J)" , Adam Miller , "Mansi M. Kasliwal" , Eric Bellm , Tom Barlow , Vandana Desai , Jason Surace , David Cook , Thomas A Prince , Yi Mei , Richard Dekany , Lin Yan , Scott Adams Cc: Steve Groom , Dave Flynn , Imel David Minutes (12 / 13 / 2016) Present: Dave S., Ben, Jason, Tom P., Frank ---- iPTF related: * Inaccessible iPTF archive products: ingestion on hold until the right contact returns from emergency leave. Timeframe is next week. Stay tuned. * Difference-image archive directory (“diffimprods”) now visible from ptf-depot web server. This can be used to support ongoing realtime scanning in January. * Request to only copy required products to ptf-depot from realtime pipeline for remainder of iPTF (Russ). * Tom B. to send IPAC a list of fields for which new reference images will be required for planned g + i experiments in January. * Frank integrating ML-feature-generation code for iPTF streak detection in realtime pipeline; want to assess performance before deploying to ZTF * Frank analyzing outputs from “findstreaks” module. Need to tune to reduce number of “obvious” false-positives being fed into machine-learned classifier. ---- ZTF development and testing: * PS1 public release expected to be next Monday (Dec 19). * source-matching / relative photometry unit testing in progress (Dave S.) -- successfully executed on replicated simulated data with encouraging results. Relative photometry appears to be consuming more memory than expected. Design of stored functions to retrieve inputs are also in progress (Russ). * Frank: new “ztf-depot” concept design / experimentation in progress. * Tom P. mentioned possibility of more NSF funding to support anticipated shortfalls in development across the project -- primarily MSIP deliverables. This includes hardware. Frank (with Steve) to communicate areas of concern. * Transition to Git / GitHub (Ben); had a chat with David Imel on account privileges and visibility to ZTF project; in progress. Regards, Frank On Dec 13, 2016, at 9:05 AM, Frank Masci wrote: Hi All, Today 1PM, KS-300. Agenda: 1. iPTF - missing and/or inaccessible products in archive [resolution in progress] - archived difference images now available through ptf-depot (since ~ Oct 2015; to complement scanning) - copying of selected realtime pipeline products to ptf-depot. Not all. - request (from Tom B?): get list of specific fields that will be used for g + i experiments in Jan. - resumption of streak-classification (ML) for iPTF as lead-in to ZTF [integration/testing in progress] - done: reprocessed star-flat experiment nights using flats derived by Jakob Nordin (ZTF assessment study). 2. ZTF pipeline development status - simulating galactic plane data (waiting for PS1 release) - R&D in filtering of extractions from ZOGY outputs. - source matching/relative photometry updates - ztf-depot — design in progress - schedule for transitioning to Git / GitHub with visibility to ZTF project. --- Goto Meeting: 1. Please join my meeting. https://global.gotomeeting.com/join/595527485 2. Use your microphone and speakers (VoIP) - a headset is recommended. Or, call in using your telephone. Dial +1 (213) 493-0622 Access Code: 595-527-485 Audio PIN: Shown after joining the meeting Meeting ID: 595-527-485 From: Frank Masci Subject: Re: iPTF/ZTF Meeting Today Date: December 6, 2016 at 3:07:08 PM PST To: Russ Laher , David Shupe , Ben Rusholme , Frank Masci , Carrie Nugent , "Rebbapragada, Umaa D (398J)" , Adam Miller , "Mansi M. Kasliwal" , Eric Bellm , Tom Barlow , Vandana Desai , Jason Surace , David Cook , Thomas A Prince , Yi Mei , Richard Dekany , Lin Yan , Scott Adams Cc: Steve Groom , Dave Flynn , Imel David Minutes (12 / 06 / 2016) Present: Dave S., Ben, Jason, Russ, Scott A., Frank, Tom B. ---- iPTF related: * Inaccessible iPTF archive products: ingestion on hold until the right contact returns from emergency leave. Stay tuned. * Scott Adams (Cahill postdoc) thinks i-band fringe correction not necessary at this time; will explore more i-band flatfield data in near future. * Request to make difference-image archive directory (“diffimprods”) visible from ptf-depot web server. Russ to talk to Anastasia or submit ISG ticket. * Request to clean up ptf-depot and only copy required products from realtime pipeline for remainder of iPTF (Russ). * Tom B. to send Russ a list of fields for which new reference images will be required for planned realtime experiments in late Dec/Jan. * Frank still integrating ML-feature-generation code for iPTF streak detection in realtime pipeline; want to assess performance before deploying to ZTF; need analysis to reduce bogus rate upstream. ---- ZTF development and testing: * Brief discussion on how to simulate galactic-plane images for throughput testing (Jason); will wait for PS1 release on Dec 16. * in progress: source-matching / relative photometry (Dave S.) -- unit test case successfully executed on simulated data with encouraging results for runtime. * Russ analyzed homogeneity in Gaia catalog across provisional ZTF test fields: found that some fields/readout-channels have holes in Gaia coverage. * P48 - IPAC data transfer testing (Ben): iperf data showed a sudden drop in transfer rate last week (~ 60 Mbit/sec); HPWREN team discovered a water problem and updated a router. Will monitor again soon. * Frank: new “ztf-depot” specs for delivering realtime products were distributed. * Transition to Git / GitHub for all ZTF pipeline code (Ben); chat with David Imel on account privileges and visibility to ZTF project. Regards, Frank On Dec 6, 2016, at 8:32 AM, Frank Masci wrote: Hi All, Today 1PM, KS-300. Agenda: 1. iPTF - missing and/or inaccessible products in archive [resolution in progress] - i-band updates, references etc. - resumption of streak-classification (ML) for iPTF as lead-in to ZTF [integration/testing in progress] - reprocessing of star-flat experiment data using new flats derived from Jakob Nordin (another ZTF assessment study). 2. ZTF pipeline development status - simulating galactic plane data - data transfers (Cahill/lab - ipac) - source matching/relative photometry update - ztf-depot -- new specs distributed (event stream metadata and QA metrics) --- Goto Meeting: 1. Please join my meeting. https://global.gotomeeting.com/join/595527485 2. Use your microphone and speakers (VoIP) - a headset is recommended. Or, call in using your telephone. Dial +1 (213) 493-0622 Access Code: 595-527-485 Audio PIN: Shown after joining the meeting Meeting ID: 595-527-485 Regards, Frank From: Frank Masci Subject: Re: iPTF/ZTF Meeting Today Date: November 29, 2016 at 9:01:08 PM PST To: Russ Laher , David Shupe , Ben Rusholme , Frank Masci , Carrie Nugent , "Rebbapragada, Umaa D (398J)" , Adam Miller , "Mansi M. Kasliwal" , Eric Bellm , Tom Barlow , Vandana Desai , Jason Surace , David Cook , Thomas A Prince , Yi Mei , Richard Dekany , Lin Yan , Scott Adams Cc: Steve Groom , Dave Flynn , Imel David Minutes (11 / 29 / 2016) Present: Dave S., Eric, Ben, Jason, Russ, Scott A., Frank, Adam M. * Some iPTF products are not accessible from archive: resolution is in progress by IRSA; a member of their team who oversees deliveries is currently on emergency leave. Stay tuned. * iPTF nightly (frameproc) pipeline can now photometrically calibrate i-band data (Russ). * Scott Adams (Cahill postdoc) has offered to write a “fringe-correction” module for i-band. This will be integrated into the iPTF nightly and realtime pipelines, ready for i-band observations planned in late Dec. * new reference images were created for PTF fields containing N>=5 (g or R) exposures as of 1 week ago (Russ). * streak classification work for iPTF; in progress by Frank; goal is to retrain machine-learned classifier and understand performance before deploying for ZTF. * request to increase timespan for retaining “realtime" difference-image products on ptf-depot, perhaps to two months (driven by programs early next year). Alternative is to enable access to archived difference-image products. Frank to explore options. * ZTF development and testing: — new round of image simulations were successfully ingested and processed end-to-end (Russ). Pipeline tweaks also made by Dave and Frank. — more QA metrics identified for astrometric calibration (Dave S.) — DB-stored functions to collect QA metrics to support external QA monitoring; done (Russ) — in progress: source-matching / relative photometry (Dave S.) -- unit test case successfully executed on simulated data. — Frank updated instrumental calibration module to write more metadata to headers of PSF-catalog files to support source-matching. — “ztf-depot”: draft of pipeline/marshal interface specs distributed (Frank); met with stakeholders; feedback was received and new version of specs will be distributed soon. — Transition to Git / GitHub for all ZTF pipeline code: in progress (Ben, all). Regards, Frank On Nov 29, 2016, at 8:46 AM, Frank Masci wrote: Hi All, Today 1PM, KS-300. Agenda: 1. iPTF - missing and/or inaccessible products in archive [resolution in progress] - pipeline updates for absolute photometric calibration of i-band data - i-band fringe correction module, to prep for upcoming i-band observations and assess for ZTF [Scott Adams] - resumption of streak-classification (ML) for iPTF as lead-in to ZTF [integration/testing in progress] - reprocessing of star-flat experiment data using new flats derived from Jakob Nordin (another ZTF assessment study). 2. ZTF pipeline development status - data transfers (Cahill/lab - ipac). - conversion of IDL coverage-map code - source matching/relative photometry update - pushing selected metrics for QA monitoring - ztf-depot -- marshal interface specs; received feedback from project [new version in prep] --- Goto Meeting: 1. Please join my meeting. https://global.gotomeeting.com/join/595527485 2. Use your microphone and speakers (VoIP) - a headset is recommended. Or, call in using your telephone. Dial +1 (213) 493-0622 Access Code: 595-527-485 Audio PIN: Shown after joining the meeting Meeting ID: 595-527-485 Regards, Frank From: Frank Masci Subject: Re: iPTF/ZTF Meeting Today Date: November 15, 2016 at 4:04:04 PM PST To: Russ Laher , David Shupe , Ben Rusholme , Frank Masci , Carrie Nugent , "Rebbapragada, Umaa D (398J)" , Adam Miller , "Mansi M. Kasliwal" , Eric Bellm , Tom Barlow , Vandana Desai , Jason Surace , David Cook , Thomas A Prince , Yi Mei , Richard Dekany , Lin Yan Cc: Steve Groom , Dave Flynn , Imel David Minutes (11 / 15 / 2016) Present: Dave S., Eric, Ben, Jason, Russ, Frank * an archive disk partition filled up last weekend. This affected the archiving of ~ two nights worth of H-alpha data. Data was recovered from a backup disk by Russ. Ron is currently re-ingesting, processing, and archiving this data. * reprocessing of PTF nights not previously archived: a few nights remain (in progress by Ron) * reprocessing/re-archiving of nights (~ Sep 23 -- Oct 22 / 2016) due to previously broken absolute photometric calibration: almost done; a few nights still remaining (in progress). * pipeline updates needed to support near-future i-filter observations with iPTF; primarily absolute photometric calibration (Russ -- in progress). We expect i-filter observations around mid/late December. The intent is to create reference images to feed the realtime pipeline for data coming in late Jan/early Feb. The goal is to understand i-filter observations for ZTF. * there’s a request to inventory all PTF fields containing N>=5 (g or R) exposures but with missing reference images; intent is to create references for these fields. * resumption of streak classification work for iPTF; in progress by Frank; goal is to understand performance before deploying for ZTF. * ZTF development and testing: — ingestion of Gaia catalogs into dev DB done; ready for analysis; in particular, to identify holes/non-uniformites over sky that may be challenging for astrometric calibration. — new round of image simulations successfully ingested (Russ). Pipeline tweaks needed before proceeding with processing (Dave, Frank). — more QA metrics identified for astrometric calibration (Dave S., in progress) — DB-stored functions to collect QA metrics to support external QA monitoring; done (Russ) — in progress: new source-matching / matchfile generation infrastructure (Dave S.); Frank to update script to write metadata keywords to header of PSF-catalog file. — “ztf-depot”: draft of pipeline/marshal interface specs were distributed (Frank) — Transition to Git / GitHub for all ZTF pipeline code: in progress (Ben, all); Eric made suggestion of periodically synching with ZTF-collaboration GitHub account (to be explored). Regards, Frank From: Frank Masci Subject: Re: iPTF/ZTF Meeting Today Date: November 8, 2016 at 2:12:45 PM PST To: Russ Laher , David Shupe , Ben Rusholme , Frank Masci , Carrie Nugent , "Rebbapragada, Umaa D (398J)" , Adam Miller , "Mansi M. Kasliwal" , Eric Bellm , Tom Barlow , Vandana Desai , Jason Surace , David Cook , Thomas A Prince , Yi Mei , Richard Dekany , Lin Yan Cc: Steve Groom , Lisa Storrie-Lombardi , Dave Flynn Minutes (11 / 08 / 2016) Present: Dave S., Eric, Ben, Jason, Adam, Frank * reprocessing of PTF nights not previously archived: Done… waiting for response from Wei on successful ingestion of metadata. Frank to check. * reprocessing/re-archiving of nights (~ Sep 23 -- Oct 22 / 2016) due to previously broken absolute photometric calibration: about half are done. Here are the remaining nights: Oct 7-13, 20-22. * pipeline updates needed to support near-future i-filter observations with iPTF; primarily absolute photometric calibration (Russ -- in progress). Frank to check with Tom B. on when to expect i-band data. * special reference images were generated (Russ) to support previous IceCube triggers * resumption of streak classification work for iPTF; in progress by Frank; goal is to understand performance before deploying for ZTF. * ZTF development and testing: — new round of image simulations successfully ingested and being processed by realtime pipeline (Russ). These include g+r filters, streaks, and span a longer night. — Ingestion of Gaia catalogs into dev DB for analysis (Russ, in progress). — Collection of QA metrics from ops DB to support external QA monitoring; almost complete (Russ) — in progress: new source-matching / matchfile generation infrastructure (Dave S.) — Star/Galaxy catalogs from PTF Transients DB still being ingested into ZTF DB (Russ) — “ztf-depot” -- marshal interface specs for delivering realtime products: in progress (Frank) — Transition to Git / GitHub for all ZTF pipeline code: in progress (Ben, all) Regards, Frank From: Frank Masci Subject: Re: iPTF/ZTF Meeting Today Date: November 1, 2016 at 5:44:13 PM PDT To: Russ Laher , David Shupe , Ben Rusholme , Frank Masci , Carrie Nugent , "Rebbapragada, Umaa D (398J)" , Adam Miller , "Mansi M. Kasliwal" , Eric Bellm , Tom Barlow , Vandana Desai , Jason Surace , David Cook , Thomas A Prince , Yi Mei , Richard Dekany , Lin Yan Cc: Steve Groom , Lisa Storrie-Lombardi , Dave Flynn Minutes (11 / 01 / 2016) Present: Dave S., Eric, Russ, Ben, Jason, Adam, Frank * relative photometry updates: ops DB needs to be updated with ZP corrections (pending, Dave S.) * reprocessing of PTF nights not previously archived: Done… waiting for response from Wei on successful ingestion of metadata. Frank to check. * absolute photometric calibration for iPTF broken since 9/23 (related to SDSS server updates). Data will be reprocessed and re-archived — in progress by Ron. * pipeline updates needed to support near-future i-filter observations with iPTF; primarily absolute photometric calibration (Russ - in progress) * resumption of streak classification work for iPTF; in progress by Frank; goal is to understand performance before deploying for ZTF. * request for new g-band reference images by Mansi; field selection in progress. * update of IDL code to generate coverage maps for two new (most recent) Halpha filters (Dave Cook to check; Jason also to check what exactly is being done for Halpha). * ZTF development and testing: — new round of image simulations successfully ingested and being processed by realtime pipeline (Russ). These include g+r filters, streaks, and span a longer night. Runtimes are longer due to new functionality. — Ingestion of Gaia catalogs into dev DB for analysis (Russ, in progress). — DB queries to collect QA metrics from ops DB to support external QA monitoring; in progress (Russ) — in progress: new source-matching / matchfile generation infrastructure (Dave S.) — ZOGY image subtraction code now fully integrated with new PSF-fitter (Frank) — prototyping canned SQL queries for querying Transients DB and pushing to ztf-depot (Frank) — revision of schema for “Stars” table (Frank). — Star/Galaxy catalogs from PTF Transients DB still being ingested into ZTF DB (Russ) — Xpra desktop management tool now installed for remote access to ztf nodes (Ben) Regards, Frank On Nov 1, 2016, at 9:03 AM, Frank Masci wrote: Hi All, Today 1PM, KS-300. Agenda: 1. iPTF - DB updates for recent relative-phot run - reprocessed nights not previously archived? - reprocessing of recent nights due to broken absolute photometric calibration (from ~ 9/17 to 10/17) - updates for absolute photometric calibration of i-band data - resumption of streak-classification (ML) for iPTF as lead-in to ZTF. 2. ZTF pipeline development status - ingestion of new simulations; - data transfers (Cahill/lab - ipac). - conversion of IDL coverage-map code; - source matching/lightcurve DB; - matched-filtering code now in realtime PL. - Gaia catalog analysis; - pushing selected metrics for QA monitoring; - experimental sql queries for transients (Stars table prep). From: Frank Masci Subject: Re: iPTF/ZTF Meeting Today Date: October 25, 2016 at 4:33:39 PM PDT To: Russ Laher , David Shupe , Ben Rusholme , Frank Masci , Carrie Nugent , "Rebbapragada, Umaa D (398J)" , Adam Miller , "Mansi M. Kasliwal" , Eric Bellm , Tom Barlow , Vandana Desai , Jason Surace , David Cook , Thomas A Prince , Yi Mei , Richard Dekany , Lin Yan Cc: Steve Groom , Lisa Storrie-Lombardi , Dave Flynn Minutes (10 / 25 / 2016) Present: Dave S., Eric, Russ, Ben, Jason, Adam, Frank * relative photometry updates: ops DB needs to be updated with ZP corrections (pending, Dave S.) * reprocessing of PTF nights not previously archived: in progress by Ron * absolute photometric calibration for iPTF broken since 9/23 (related to SDSS server updates). Data will be reprocessed and re-archived — also in progress by Ron. * pipeline updates needed to support near-future i-filter observations with iPTF; primarily absolute photometric calibration (Russ) * resumption of streak classification work for iPTF; funds for JPL now flowing; IPAC needs to deploy new feature-generation code from Brian (Frank & Russ); training set will then be constructed; goal is to understand performance before deploying for ZTF. * iPTF star-flat experiment: intermediate pipeline products were forwarded to Jakob (Frank) * ZTF development and testing: — new round of image simulations now ready for ingestion. These now include g+r, streaks, and span a longer night. — Gaia catalogs were ingested into a dev DB for analysis (Russ). Goal is to explore uniformity at the sub readout-channel. — instrumental calibration pipeline now using Gaia to astrometrically calibrate sim images. — astrometric calibration fixes for cases with “bad” input WCS (Dave S.) — DB queries to collect QA metrics from ops DB to support external QA monitoring; in progress (Russ) — in progress: new source-matching / matchfile generation infrastructure (Dave S.). — prototyping of new image-subtraction code (ZOGY method) and transient detection algorithm (Frank) — prototyping canned SQL queries for querying Transients DB; revision of schema for “Strars” table (Frank) — request to install a VNC (or NX) server on ZTF node for faster access to X-windows apps (pending, Ben). Regards, Frank From: Frank Masci Subject: Re: iPTF/ZTF Meeting Today Date: October 18, 2016 at 3:46:55 PM PDT To: Russ Laher , David Shupe , Ben Rusholme , Frank Masci , Carrie Nugent , "Rebbapragada, Umaa D (398J)" , Adam Miller , "Mansi M. Kasliwal" , Eric Bellm , Tom Barlow , Vandana Desai , Jason Surace , David Cook , Thomas A Prince , Yi Mei , Richard Dekany , Lin Yan Cc: Steve Groom , Lisa Storrie-Lombardi , Dave Flynn Minutes (10 / 18 / 2016) Present: Dave S., Eric, Russ, Tom B., Jason, Frank * relative photometry updates: ops DB needs to be updated with ZP corrections (Dave S.) * reprocessing of PTF nights not previously archived: in progress by Ron * absolute photometric calibration for iPTF broken since 9/23 (related to SDSS server updates). Data will be reprocessed and re-archived. * new g-band reference images in progress (request from Mansi) * pipeline updates needed to support near-future i-filter observations with iPTF; primarily absolute photometric calibration (Russ) * iPTF star-flat experiment: intermediate pipeline products were forwarded to Jakob (Frank) * ZTF development and testing: — sim-schedule file needed by Jason to generate next round of simulations (Eric) — generated readout-channel partitioned Gaia catalogs for use in astrometric calibration (Russ); used a corner-coordinate file generated from Eran’s sky tiling (Frank). — desire to start using Gaia astrometric catalogs when processing simulated data, not SDSS (Russ to ensure) — in progress: exploring degree of uniformity of Gaia catalogs on sub-readout-channel scales to ensure astrometry can be adequately calibrated (Dave S.). Goal is to isolate possible problematic ZTF fields. — collected QA metrics from ops DBs to support external QA monitoring; added to Thomas’ specs document (Frank) — fixed some bugs in asteroid association software ("astcheck") in realtime pipeline (Frank) — in progress: new source-matching / matchfile generation infrastructure (Dave S.). — Request to install a VNC (or NX) server on ZTF node for faster access to X-windows apps (pending, Ben). Regards, Frank From: Frank Masci Subject: Re: iPTF/ZTF Meeting Today Date: October 11, 2016 at 4:03:27 PM PDT To: Russ Laher , David Shupe , Ben Rusholme , Frank Masci , Carrie Nugent , "Rebbapragada, Umaa D (398J)" , Adam Miller , "Mansi M. Kasliwal" , Eric Bellm , Tom Barlow , Vandana Desai , Jason Surace , David Cook , Thomas A Prince , Yi Mei , Richard Dekany , Lin Yan Cc: Steve Groom , Lisa Storrie-Lombardi , Dave Flynn Minutes (10 / 11 / 2016) Present: Dave S., Russ, Tom B., Jason, Adam, Frank * source-matching/relative photometry updates: ops DB needs to be updated with ZP corrections (Dave S.) * iPTF forced photometry server/job-dispatcher restarted and accepting jobs again. * reprocessing of PTF nights not previously archived: in progress by Ron Beck. * pipeline updates needed to support future i-filter observations with iPTF, primarily absolute photometric cal. (Russ). * ZTF development and testing: — sim-schedule file need by Jason to generate next round of simulations (Eric). — Updated real-time pipeline and DB schema to associate transients with known solar-system objects (Frank, Russ). — Generated first round of Gaia catalogs for use in astrometric calibration in pipeline (Russ); lots of timeouts seen; Frank to check with IRSA on other faster query protocols, e.g., TAP service; suggestion to filter high proper-motion sources (Dave S?). — Planning of new source-matching / matchfile generation infrastructure (in progress: Dave S. and Ben). — Request to install a VNC (or NX) server on ZTF node for faster access to X-windows apps (pending, Ben). Regards, Frank From: Frank Masci Subject: Re: iPTF/ZTF Meeting Today Date: October 4, 2016 at 3:59:36 PM PDT To: Russ Laher , David Shupe , Ben Rusholme , Frank Masci , Carrie Nugent , "Rebbapragada, Umaa D (398J)" , Adam Miller , "Mansi M. Kasliwal" , Eric Bellm , Tom Barlow , Vandana Desai , Jason Surace , David Cook , Thomas A Prince , Yi Mei , Richard Dekany , Lin Yan Cc: Steve Groom , Lisa Storrie-Lombardi , Dave Flynn Minutes (10 / 04 / 2016) Present: Dave S., Ben, Russ, Jason, Adam, Frank * source-matching/relative photometry for PTF up to date. Ops DB needs to be updated with ZP corrections (Dave S.) * reprocessing of nights not previously archived: in progress by Ron Beck. * PTF realtime pipeline was run for ~10 fields from early July. These were not processed first time round (high cadence gal-plane fields). * discussion on whether PTF processing can handle i-band data (requested by Mansi to help with decisions for ZTF)…; Russ: almost if some tweaks are made to the absolute calibration code. Realtime pipeline may also need work -- requires i-band reference images. Frank to check with Tom B. on scheduling of i-band observations. * special PTF fields were observed this week to generate “star-flats" -- will help with flat-fielding decisions for ZTF. Collaboration member (Jakob) was notified. * ZTF development and testing: — sim-schedule file in preparation (Eric) to use in next round of simulations (for Jason). Simulate night with 12 repeats per field (6 in g, 6 in R), falling at/near Winter solstice. — Updated ops DB schema to store more QA metrics (specifically from astrometric calibration). — Updated real-time pipeline and DB schema to associate transients with known solar-system objects (Frank, Russ. — Planning of new source-matching / matchfile generation infrastructure (in progress: Dave S. and Ben). — Conversion of coverage-map generation software from IDL to Python in progress (Ben). — Request to install a VNC (or NX) on a ZTF node for faster access to X-windows applications outside building (Ben). Regards, Frank On Oct 4, 2016, at 9:12 AM, Frank Masci wrote: Hi All, Today 1PM, KS-300. Agenda: 1. iPTF - source-matching/relative-phot update - reprocessed nights not being archived? - rerun of realtime pipeline for high-cadence g-plane fields - prep for upcoming i-band experiments - star-flat experiment - ZTF-shutter mode analysis 2. ZTF pipeline development status (simulations; QA metrics; conversion of IDL coverage-map code; image-differencing; source matching/lightcurve DB; Gaia catalogs). --- Goto Meeting: 1. Please join my meeting. https://global.gotomeeting.com/join/595527485 2. Use your microphone and speakers (VoIP) - a headset is recommended. Or, call in using your telephone. Dial +1 (213) 493-0622 Access Code: 595-527-485 Audio PIN: Shown after joining the meeting Meeting ID: 595-527-485 rom: Frank Masci Subject: Re: iPTF/ZTF Meeting Today Date: September 20, 2016 at 3:13:26 PM PDT To: Russ Laher , David Shupe , Ben Rusholme , Frank Masci , Carrie Nugent , "Rebbapragada, Umaa D (398J)" , Adam Miller , "Mansi M. Kasliwal" , Eric Bellm , Tom Barlow , Vandana Desai , Jason Surace , David Cook , Thomas A Prince , Yi Mei , Richard Dekany , Lin Yan Cc: Steve Groom , Lisa Storrie-Lombardi , Dave Flynn Minutes (09 / 20 / 2016) Present: Dave S., Ben, Russ, Eric, Jason, Frank * new round of source-matching for PTF done using existing reference images (Dave S.). 1500 (field/chip/filter) pointings had missing references; Dave to forward list to Russ so references can be generated where possible. Also, about 300 matchfiles had corruption issues. * some feedback from lightcurve DB usage: * reprocessing of PTF nights not being archived properly: night ID 1958 (8/20/14) was reprocessed and delivered to IRSA. Waiting for Wei to confirm ingest of archive metadata. Will reprocess other nights once confirmed. * analysis of data acquired using ZTF shutter (vs PTF shutter): in progress, more experiments planned this week. * ZTF development and testing: -- simulation code updated (Jason) to generate longer night consisting of multi-filter observations; also includes streaks; waiting for schedule file from Eric. -- streak detection now integrated into realtime pipeline (Frank & Russ), including new “streaks” table. More metrics with RealBogus pending. Goal is to pre-filter according to RB before loading. -- QA metrics from astrometric calibration in progress (Dave). Frank to provide more test cases. -- performance metrics on source-matching and relative photometry from PTF to help with ZTF design (Dave). -- also discussed possible new matchfile formats (Dave): SQLlite, Sparc compatibility..? -- conversion of coverage-map generation software from IDL to Python in progress (Ben). -- new code deployed for realtime pipeline to associate transient candidates with _known_ asteroids from MPC archive (Frank; in progress). -- plan to use another versioning system for ZTF software, e.g., Git (Ben). Regards, Frank From: Frank Masci Subject: Re: iPTF/ZTF Meeting Today Date: September 27, 2016 at 3:43:08 PM PDT To: Russ Laher , David Shupe , Ben Rusholme , Frank Masci , Carrie Nugent , "Rebbapragada, Umaa D (398J)" , Adam Miller , "Mansi M. Kasliwal" , Eric Bellm , Tom Barlow , Vandana Desai , Jason Surace , David Cook , Thomas A Prince , Yi Mei , Richard Dekany , Lin Yan Cc: Steve Groom , Lisa Storrie-Lombardi , Dave Flynn Minutes (09 / 27 / 2016) Present: Dave S., Ben, Russ, Eric, Frank * source-matching/relative photometry for PTF is now up to date. 1217 new reference images were made, resulting in 1288 new matchfiles. 230 matchfiles are corrupted (to explore later). Ops DB to be updated with ZP corrections. * reprocessed nights not previously archived: Frank to check with Wei on status of newly reprocessed night [night ID 1958; 8/20/14]. Will reprocess other nights once confirmed. * PTF realtime pipeline being rerun for ~10 fields from early July (Russ). These were not run during routine operations. * Special PTF fields scheduled for later this week to generate “star-flats” offline -- will help with flat-fielding decisions for ZTF. * ZTF development and testing: -- sim-schedule file in preparation (Eric) to use in next round of simulations (for Jason). Simulate night with 12 repeats per field (6 in g, 6 in R), falling at/near Winter solstice. -- new throughput timing estimate (typical realtime pipeline runtimes per readout channel are now ~ 100 sec, previously ~170 sec); 34 drones now operational. -- QA metrics from astrometric calibration: in progress (Dave). -- discussed possible new matchfile formats (Dave and Eric): it appears the concept of a matchfile is still popular. Some redesign needed. -- conversion of coverage-map generation software from IDL to Python in progress (Ben). -- new code for realtime pipeline to associate transient candidates with _known_ asteroids from MPC archive (Frank; in progress). -- plans to use another versioning system for ZTF software; GitHub is a good option (Ben). Regards, Frank From: Frank Masci Subject: Re: iPTF/ZTF Meeting Today Date: September 13, 2016 at 5:34:34 PM PDT To: Russ Laher , David Shupe , Ben Rusholme , Frank Masci , Carrie Nugent , "Rebbapragada, Umaa D (398J)" , Adam Miller , "Mansi M. Kasliwal" , Eric Bellm , Tom Barlow , Vandana Desai , Jason Surace , David Cook , Thomas A Prince , Yi Mei , Richard Dekany , Lin Yan Cc: Steve Groom , Lisa Storrie-Lombardi , Dave Flynn Minutes (09 / 13 / 2016) Present: Dave S., Ben, Russ, Eric, Tom P., Jason, Frank * new round of source-matching for PTF in progress. Some issues with missing/corrupt matchfiles (Dave S.). Eric requests products be ready within the next two weeks. * reprocessed PTF nights not being archived properly. A night that was never archived (nid=1999) was reprocessed and delivered to IRSA. Waiting for feedback. Once successful, will then inventory other nights where archive has older processed versions. * Installation of ZTF shutter on P48 is happening this week. IPAC to check that test data is all ingested and sane. Whether all/some of the on-sky data needs to be archived is TBD (Frank). * P48-IPAC throughput/bandwidth testing: Ben performed new transfer tests using randomized sim image files. Initially recovered the 20 sec transfer time (~ 150 Mbps). This increased to > 40 sec later in the night ( <~ 75 Mbps). * ZTF development and testing: -- items for new round of image sims in near future: (i) longer night consisting of multi-filter observations (Eric to deliver survey plan to Jason); (ii) include objects that streak on individual chips/quads; (iii) ensure OBJRA/DEC, TELRA,DEC and CRVAL1,2 are consistent; (iv) ensure EXTNAME is a string. -- continued throughput testing; early finding (Russ): can process ~ 8 hours worth of on-sky data in ~ 9 hours using 25 machines. -- QA metrics from astrometric calibration in progress (Dave). -- Frank started to design/integrate streak detection with QA in realtime pipeline. Tom P. suggested utilizing shutter open/close times and streak light profile to attempt to derive direction of moving object. -- Dave, Frank, Steve… starting to think about matchfile deliverables (source matching and relative photom) for ZTF. -- conversion of coverage-map generation software from IDL to Python in progress (Ben). Regards, Frank On Sep 13, 2016, at 8:55 AM, Frank Masci wrote: Hi All, Today 1PM, KS-300. Agenda: 1. iPTF requests/issues - source-matching/relative-phot - new reference images for LBNL (neutrino project at DESY) - reprocessed nights not being archived? 2. ZTF Shutter installation on P48 this week. Pipeline analysis. 3. P48-IPAC throughput/bandwidth testing update. 4. ZTF pipeline development status (new round of sims[?]; throughput testing; astrometric calibration with qa; conversion of IDL coverage-map code; image-diff; streak detection; plans for source matching and lightcurve DB). --- Goto Meeting: 1. Please join my meeting. https://global.gotomeeting.com/join/595527485 2. Use your microphone and speakers (VoIP) - a headset is recommended. Or, call in using your telephone. Dial +1 (213) 493-0622 Access Code: 595-527-485 Audio PIN: Shown after joining the meeting Meeting ID: 595-527-485 From: Frank Masci Subject: Re: iPTF/ZTF Meeting Today Date: September 6, 2016 at 3:18:51 PM PDT To: Russ Laher , David Shupe , Ben Rusholme , Frank Masci , Carrie Nugent , "Rebbapragada, Umaa D (398J)" , Adam Miller , "Mansi M. Kasliwal" , Eric Bellm , Tom Barlow , Vandana Desai , Jason Surace , David Cook , Thomas A Prince , Yi Mei , Richard Dekany , Lin Yan Cc: Steve Groom , Lisa Storrie-Lombardi , Dave Flynn Minutes (09 / 06 / 2016) Present: Dave S., Dave C., Ben, Eric, Lin, Jason, Frank * PTF Lightcurve Database feedback: request to update plot panel to (i) include error bars on photometry points; (ii) flip Y-axis so magnitude increases downward. Frank also added more cautionary notes to documentation: primarily on use of photometric errors on ZP-corrected measurements. * Halpha reprocessing: Dave C. satisfied with recalibration of catalog files for those fields/epochs that overlap with SDSS. Mention of archiving updated catalog files (for collaboration access) in the near future. Frank to work out details with IRSA. * PTF/iPTF source-matching/relative-photometry update: run by Dave S. Now exploring some missing/corrupt matchfiles (fifty-ish or so). * More iPTF requests: 15 special reference images to be constructed to support lightcurve generation on neutrino burst triggers. * Issue on missing reprocessed PTF/iPTF nights in archive: to be explored. * Lin: mentioned preparation and possible topics for ZTF collaboration meeting: Nov 30 -- Dec 2, Caltech. * ZTF development: Ben -- discussions with ISG on getting through IPAC firewall; new build infrastructure and environment; suggested setting up a regression test (RTB) using current build version. Dave -- designing and implementing QA metrics related to astrometric calibration. Frank -- unit testing of pipeline software using latest build and configuration, so far so good. Regards, Frank On Sep 6, 2016, at 8:52 AM, Frank Masci wrote: Hi All, Today 1PM, KS-300. Agenda: 1. PTF lightcurve release: status, feedback etc. 2. iPTF requests/issues - Halpha reprocessing - source-matching/relative-phot - new reference images for LBNL (neutrino project at DESY) - reprocessed nights not being archived? 3. P48-IPAC throughput/bandwidth testing update. 4. ZTF pipeline development status (new round of sims[?]; throughput testing; astrometric calibration; image-diff; plans for source matching and lightcurve DB). --- Goto Meeting: 1. Please join my meeting. https://global.gotomeeting.com/join/595527485 2. Use your microphone and speakers (VoIP) - a headset is recommended. Or, call in using your telephone. Dial +1 (213) 493-0622 Access Code: 595-527-485 Audio PIN: Shown after joining the meeting Meeting ID: 595-527-485 From: Frank Masci Subject: Re: iPTF/ZTF Meeting Today Date: August 30, 2016 at 3:02:09 PM PDT To: Russ Laher , David Shupe , Ben Rusholme , Frank Masci , Carrie Nugent , "Rebbapragada, Umaa D (398J)" , Adam Miller , "Mansi M. Kasliwal" , Eric Bellm , Tom Barlow , Vandana Desai , Jason Surace , David Cook , Thomas A Prince , Yi Mei , Richard Dekany , Lin Yan Cc: Steve Groom , Lisa Storrie-Lombardi , Dave Flynn Minutes (08 / 30 / 2016) Present: Dave S., Tom P., Ben, Eric, Russ, Adam [and his dog], Jason, Frank * Data Release Three (DR3): all appears to be set. Floodgates to open Sept 1, 00hrs PDT. New pages: http://www.ptf.caltech.edu/page/DR3 http://www.ptf.caltech.edu/page/lcdb http://www.ptf.caltech.edu/page/lcgui ---> Eric to announce to iptfgeneral and add link to PTF twiki containing password for Lightcurve DB access. * Adam M. to provide Frank with a high |b| field to explore more typical photometric performance from Lightcurve DB: Done. * Halpha update, Dave C. (via email): “Calibration is done and went well. I have transferred the outputs to my caltech computer and have begun the candidate selection process from the calibrated data. Also, I was able to successfully incorporate image cutouts from irsa via a method similar to the variable marshall." * Issue found on archiving specific PTF/iPTF nights. Inconsistency with versions in ops DB: to be explored. DR3 contents are fine. * Update to source matching/relative photometry for iPTF: five months worth of data were recently matched (Dave S.). Need to update relphotcal table in ops DB with ZP info. Eric mentioned some corrupted matchfiles; same ones may keep on failing…? * ZTF development and testing: - Ben: submitted ISG tickets for firewall access to support P48-IPAC data transfer; build infrastructure in place. - Dave S: astrometric calibration module delivered; will be integrated soon; QA metrics in progress. - Jason: suggested implementing distortion sanity checker (carry over from PTF) - Dave S. to reimplement. - Russ: preliminary throughput testing using realtime pipeline on 25 cluster nodes; new benchmarks. - Frank: optimizing the instrumental calibration pipeline; will test against new build environment. - Jason/Eric: future sim update: provide bigger/more representative set of sim images spanning a longer night, also consisting of multiple filters. Regards, Frank On Aug 30, 2016, at 9:13 AM, Frank Masci wrote: Hi All, Today 1PM, KS-300. Agenda: 1. Data Release Three: ~ 38 hours to go; Documentation -- 3 new pages on PTF website; many others updated: http://www.ptf.caltech.edu/page/DR3 http://www.ptf.caltech.edu/page/lcdb http://www.ptf.caltech.edu/page/lcgui 2. Halpha reprocessing update. 3. Ad-hoc iPTF requests: new round of source-matching/relative-phot; star-flat experiment to help with ZTF planning. 4. P48-IPAC throughput/bandwidth testing update. 5. ZTF pipeline development status (sim status; ops filesystem/testing; astrometric calibration; image-diff; plans for source matching and lightcurve DB). --- Goto Meeting: 1. Please join my meeting. https://global.gotomeeting.com/join/595527485 2. Use your microphone and speakers (VoIP) - a headset is recommended. Or, call in using your telephone. Dial +1 (213) 493-0622 Access Code: 595-527-485 Audio PIN: Shown after joining the meeting Meeting ID: 595-527-485 From: Frank Masci Subject: Re: iPTF/ZTF Meeting Today Date: August 23, 2016 at 4:49:16 PM PDT To: Russ Laher , David Shupe , Ben Rusholme , Frank Masci , Carrie Nugent , "Rebbapragada, Umaa D (398J)" , Adam Miller , "Mansi M. Kasliwal" , Eric Bellm , Tom Barlow , Vandana Desai , Jason Surace , David Cook , Thomas A Prince , Yi Mei , Richard Dekany , Lin Yan Cc: Steve Groom , Lisa Storrie-Lombardi , Dave Flynn Minutes (08 / 23 / 2016) Present: Russ, Tom P., Adam M., Dave S., Frank * Data release preparation (DR3): Frank working on online documentation and recipes for the LC retrieval interface; analysis of photometric accuracy with/without ZP-corrections done for a complex field. Results are as expected. * Adam M. to provide Frank with a high |b| field to explore more typical photometric performance from LC DB. * Another round of PTF/iPTF source matching/relative photometry requested (Dave S. to run). * Frank to ensure that any iPTF data acquired to support Gaia (under Caltech time) is not made public for DR4; to obtain program ID from Tom B. * ZTF development and testing: - Russ: preliminary throughput testing using realtime pipeline on 25 cluster nodes, exercising new ops filesystem and DBs; created a new DB table: “jobs” to track pipeline metadata (runtimes, exit codes..). - Frank: optimizing the instrumental calibration pipeline in response to hickups found in testing. - Dave S: astrometric calibration module in progress; expect to integrate soon. * Frank to convene meeting with Tom P., Eric, Dave S., and IRSA to discuss future ZTF Lightcurve DB. Regards, Frank On Aug 23, 2016, at 9:39 AM, Frank Masci wrote: Hi All, Today 1PM, KS-300. Agenda: 1. Data Release 3 preparation, Lightcurve DB status (for collaboration), documentation… 7 days to go. 2. Halpha reprocessing update. 3. Ad-hoc iPTF requests: new round of source-matching/relative-phot; star-flat experiment to help with ZTF planning. 4. P48-IPAC throughput/bandwidth testing update. 5. ZTF pipeline development status (sim status; ops filesystem/testing; astrometric calibration; image-diff; plans for source matching). --- Goto Meeting: 1. Please join my meeting. https://global.gotomeeting.com/join/595527485 2. Use your microphone and speakers (VoIP) - a headset is recommended. Or, call in using your telephone. Dial +1 (213) 493-0622 Access Code: 595-527-485 Audio PIN: Shown after joining the meeting Meeting ID: 595-527-485 Minutes (08 / 16 / 2016) Present: Ben, Dave C., Jason, Russ, Eric, Frank * PTF 2016 release status: Frank working on documentation and recipes on using the LC interface; analysis of photometric accuracy with/without ZP-corrections in progress. * Dave C. reported good progress on reprocessing (recalibration) of Halpha image and catalog data, primarily that which overlaps with SDSS. * Eric requested another round of source-matching/relative-photometry for iPTF. Last run was in mid March. * Ben presented new bandwidth test results using real same-night transfers of 960 ZTF-like exposures (concurrent with iPTF Halpha observations in progress). Median bandwidth is ~148 Mbps; 10-percentile is ~113 Mbps; 1-percentile is ~64 Mbps. Suggestions: transfer a mix of simulated image files instead of the same one; look for any correlations/clustering in arrival times, particularly for tail cases. Jason: maybe possibility of making the compression lossless (instead of lossy like now) -- too early to tell. * Frank to look at iPTF image transfer times from same nights as used by Ben to see how things compare. * Future ZTF image simulator upgrade: suggestion made to simulate more than one filter (Eric to provide schedule file); also simulate an entire night’s worth of image data. * ZTF development and testing: - Russ running the realtime (primarily instrumental calibration) pipeline using the new operations filesystem, database, and 25 nodes (for now); also tested the reference-image pipeline; a run that includes image-subtraction is in progress. - Ben reorganized the build infrastructure and rebuilt software under operations filesystem; installed latest versions of external libraries. Regards, Frank On Aug 16, 2016, at 8:36 AM, Frank Masci wrote: Hi All, Today 1PM, KS-300. Agenda: 1. PTF/iPTF lightcurve retrieval interface, feedback from summer school, analysis and to-do list. 2. Halpha reprocessing update. 3. P48-IPAC throughput/bandwidth testing update (relax compression?) 4. ZTF pipeline development status (ops filesystem/testing; astrometric calibration; image-diff; plans for source matching). Regards, Frank From: Frank Masci Subject: Re: iPTF/ZTF Meeting Today Date: August 2, 2016 at 4:04:20 PM PDT To: Russ Laher , David Shupe , Ben Rusholme , Frank Masci , Carrie Nugent , "Rebbapragada, Umaa D (398J)" , Adam Miller , "Mansi M. Kasliwal" , Eric Bellm , Tom Barlow , Vandana Desai , Jason Surace , David Cook , Thomas A Prince , Yi Mei , Richard Dekany , Lin Yan Cc: Steve Groom , Lisa Storrie-Lombardi Minutes (08 / 02 / 2016) Present: Ben, Dave S., Dave C., Jason, Russ, Frank * PTF 2016 release status: Frank working on documentation and recipes on using the LC interface; analysis of photometric accuracy (i.e., performance of relative photometry) pending. * New point-source realbogus model was deployed to iPTF operations on 7/20/2016. * Halpha reprocessing: WCS update and recalibration plan in place (Dave C.); no more than 1TB of sandbox disk space needed for now; Dave S. to review correct WCS-fixing for Dave C. * Pending: issue with IPAC-table uploads to query Halpha data through IRSA: both using GUI and API. Currently being looked at by IRSA. * ZTF throughput/bandwidth testing from P48-IPAC. Ben reported a mode of ~ 20 sec in the transfer time of individual raw compressed CCD image files; 95% upper percentile is ~ 32 sec. This translates to ~ 160-180 mbs. Very good news. Next step is to perform a longer stress test during a “quiet” observing night. * Discussion on definition of night Id (nid) for ZTF. Need to ensure calibration and survey data are assigned the same nid. Russ will look into definition and ensure assignment is "noon Pacific Time" to "noon Pacific Time next day”. * Ben had a chat with Reed Riddle on ZTF data transfer protocols from P48-IPAC; Reed prefers to push, while for security reasons, Eugean from IPAC ISG recommends pulling. Interface details to be discussed. Reed: schedule to get prototype software ready for testing at P48 is late Sep/early Oct. * ZTF pipeline development: - new operations filesystem started to be exercised (Russ and Frank) - new astrometric calibration module in progress (Dave S.) - new image-differencing module delivered by Brad Cenko; currently being checked by Eran Ofek. Regards, Frank On Aug 2, 2016, at 9:53 AM, Frank Masci wrote: Hi All, Today 1PM, KS-300. Agenda: 1. PTF/iPTF lightcurve retrieval interface, feedback from summer school, analysis and to-do list. 2. New point-source realbogus for remainder of iPTF now operational. 3. Halpha reprocessing update. 4. P48-IPAC throughput/bandwidth testing update (relax compression?) 5. ZTF pipeline development status (ops filesystem/testing; astrometric calibration; image-diff; plans for source matching). From: Frank Masci Subject: Re: iPTF/ZTF Meeting Today Date: June 28, 2016 at 3:06:32 PM PDT To: Russ Laher , David Shupe , Ben Rusholme , Frank Masci , Carrie Nugent , "Rebbapragada, Umaa D (398J)" , Adam Miller , "Mansi M. Kasliwal" , Eric Bellm , Tom Barlow , Vandana Desai , Jason Surace , David Cook , Thomas A Prince , Yi Mei , Richard Dekany , Lin Yan Cc: Steve Groom , Lisa Storrie-Lombardi Minutes (6 / 28 / 2016) Present: Ben, Adam, Dave S., Dave C., Umaa, Jason, Russ, Frank * PTF 2016 release preparation status: Frank working on documentation and data-dictionary/column definitions for the different table views in Gator. * Small change in PTF 2016 data release schedule: - file products (images and catalogs) will be made public on Sep 1, 2016. - Lightcurve DB and GUI will be made visible only to the PTF/iPTF collaboration on Sep 1, 2016. - A version of the lightcurve DB will be available for students at PTF/iPTF Summer School in July. - The Lightcurve DB and GUI will then become public on Dec 1, 2016. * Adam to provide a test field/ccd to check lightcurve DB. Will be same field/ccd used for demo'ing at upcoming Summer School. * new point-source realbogus model for iPTF to be delivered to IPAC either tomorrow or Thursday; will likely be integrated into realtime pipeline after July 4th * iPTF tasks in progress: CGI script updates to support realtime scanning (e.g., include cross-match to refimallsources DB table); low priority: deploy new SDSS stars table in Transients DB. * Halpha reprocessing: Dave C. to provide Russ with recalibration script when ready. * TBD: whether to deploy Dave S.’s new astrometric calibration code into the photometric pipeline. Will impact image header contents. Want to avoid changing header structure at this time from what’s already been archived. * P48-IPAC throughput/bandwidth testing: Ben reported that the bandwidth has suddenly dropped back to ~ 80 abs; it was ~175 mbs last week? * ZTF sims: jason generated a single nights worth of image data; Russ found an issue with a date-time stamp; this was solved by Jason and sims will be regenerated at end of this week. * ZTF pipeline dev: Russ integrating instrumental calibration pipeline (= first half of realtime pipeline); Frank working on image-subtraction pipeline (preparatory steps). * Dave S. to work on standalone script for ZTF astrometric calibration according to specs provided by Frank. * Starting thinking about source-matching/relative-photometry infrastructure for ZTF (Dave and Ben). Goal is to reuse software from iPTF; small issue is that only PSF-fit catalogs will be absolutely calibrated for ZTF. Photometric PSF-based ZPs will also apply to ~ big apertures from SExtractor; unclear if should continue to use SExtractor inputs for source-matching. Regards, Frank On Jun 28, 2016, at 8:11 AM, Frank Masci wrote: Hi All, Today 1PM, KS-300. Agenda: 1. PTF/iPTF Fall release update: lightcurve DB; other products; documentation. 2. Status of new point-source realbogus for remainder of iPTF. 3. Halpha reprocessing update. 4. Should new astrometric code be deployed for iPTF photometric pipeline? Issues TBR. 5. P48-IPAC throughput/bandwidth testing update. 6. ZTF simulations. 7. ZTF pipeline development status (astrometric calibration; plans for source matching). --- Goto Meeting: 1. Please join my meeting. https://global.gotomeeting.com/join/595527485 2. Use your microphone and speakers (VoIP) - a headset is recommended. Or, call in using your telephone. Dial +1 (213) 493-0622 Access Code: 595-527-485 Audio PIN: Shown after joining the meeting Meeting ID: 595-527-485 From: Frank Masci Subject: Re: iPTF/ZTF Meeting Today Date: June 21, 2016 at 3:14:04 PM PDT To: Russ Laher , David Shupe , Ben Rusholme , Frank Masci , Carrie Nugent , "Rebbapragada, Umaa D (398J)" , Adam Miller , "Mansi M. Kasliwal" , Eric Bellm , Tom Barlow , Vandana Desai , Jason Surace , David Cook , Thomas A Prince , Yi Mei , Richard Dekany , Lin Yan Cc: Steve Groom , Lisa Storrie-Lombardi Minutes (6 / 21 / 2016) Present: Ben, Umaa, Jason, Russ, Tom B., Frank * PTF release preparation status: light curve DB tables ingested and indexed in IRSA; now working on "lightcurve-table view” and optimization on the backend. * Status of new point-source realbogus for remainder of iPTF: Umaa gave a status update. Currently in progress. * New iPTF fields were defined over the weekend to support Halpha observations. These field IDs were created in the DB and the data is currently being processed. Russ reported there may be an issue with scamp accessing astrometric reference sources for these new fields. Will followup with Ron. * Reprocessing of iPTF data for interval May 11 - June 9, 2016 to fix an absolute photometric calibration issue is more than half done. This data will be re-archived. * Adam M. provided a new SDSS table file be loaded into the iPTF transients DB. Russ waiting for Adam’s reply on specific cuts to apply. * Russ also given a request to update a CGI script to facilitate more optimal realtime scanning for iPTF. * Ben to check with Tom B. on when to perform a bandwidth/data-transfer stress test from P48 to IPAC. Frank to explore whether more noise-bits can be retained during compression. * Status of new simulated ZTF images: in progress -- Eric to provide survey pointings in format requested by Jason. * ZTF development status: science, calibration, and reference-image pipeline scripts in place -- to be integrated with DBs. Frank working on the image-subtraction / transient-extraction pipeline, at least the preparatory steps. Regards, Frank On Jun 21, 2016, at 8:35 AM, Frank Masci wrote: Hi All, Today 1PM, KS-300. Agenda: 1. PTF/iPTF Fall release update: lightcurve DB; object table column definitions -- need to finalize soon. 2. Status of new point-source realbogus for remainder of iPTF. 3. Halpha reprocessing update. 4. P48-IPAC throughput/bandwidth testing update. 5. ZTF simulation update (need sample of survey pointings, ~10 overlapping epochs) 6. ZTF pipeline development status. --- Goto Meeting: 1. Please join my meeting. https://global.gotomeeting.com/join/595527485 2. Use your microphone and speakers (VoIP) - a headset is recommended. Or, call in using your telephone. Dial +1 (213) 493-0622 Access Code: 595-527-485 Audio PIN: Shown after joining the meeting Meeting ID: 595-527-485 From: Frank Masci Subject: Re: iPTF/ZTF Meeting Today Date: June 14, 2016 at 4:06:54 PM PDT To: Russ Laher , David Shupe , Ben Rusholme , Frank Masci , Carrie Nugent , "Rebbapragada, Umaa D (398J)" , Adam Miller , "Mansi M. Kasliwal" , Eric Bellm , Tom Barlow , Vandana Desai , Jason Surace , David Cook , Thomas A Prince , Yi Mei , Richard Dekany , Lin Yan Cc: Steve Groom , Lisa Storrie-Lombardi Minutes (6 / 14 / 2016) Present: Adam, Ben, Jason, Eric, Russ, Frank * PTF release preparation status: light curve DB tables reloaded in IRSA; single table views are available to query using Gator (internal access only). Request to Eric to check DD for objects table provided by Frank. Sky-coverage maps for DR3 were also made by Jason (for ref and epochal images). * Adam requesting that a new SDSS table be loaded into the iPTF transients DB to optimize realtime discoveries. Plan is to remove the existing sdss_r, sdss_g tables and create a new one: “sdss”. This will be used in conjunction with the refimg_r table to enable more complete transient-to-star associations. * Reprocessing of iPTF data for interval May 11 - June 9, 2016 currently in progress to fix an absolute photometric calibration issue. This data will be re-archived. * Ben reported a dramatic improvement in throughput when transferring ZTF-like images from P48 to IPAC. A factor of 2.5x above requirement! Will test to explore nightly sustainability and variability. Once understood, Frank to explore whether more noise-bits can be retained during compression. * Request for a new set of simulated ZTF images that spans ~10 overlapping observation epochs. The region must fall in the SDSS footprint. Also request for ~100 bias exposures and ~100 “illuminated” exposures for purpose of constructing flat fields. Eric to forward Jason list of fields to simulate, in desired format. Regards, Frank On Jun 14, 2016, at 9:23 AM, Frank Masci wrote: Hi All, Today 1PM, KS-300. Agenda: 1. PTF/iPTF Fall release update: lightcurve DB; object table column definitions (need by 6/17); coverage maps done (thanks Jason), other documentation.. 2. Status of new point-source realbogus for remainder of iPTF. 3. Halpha reprocessing. 4. New results from P48-IPAC throughput/bandwidth testing, implications for compression. 5. ZTF simulations (fixes, request for larger set, overlapping fields, etc.) 6. ZTF pipeline development status. --- Goto Meeting: 1. Please join my meeting. https://global.gotomeeting.com/join/595527485 2. Use your microphone and speakers (VoIP) - a headset is recommended. Or, call in using your telephone. Dial +1 (213) 493-0622 Access Code: 595-527-485 Audio PIN: Shown after joining the meeting Meeting ID: 595-527-485 From: Frank Masci Subject: Re: iPTF/ZTF Meeting Today Date: June 7, 2016 at 5:28:29 PM PDT To: Russ Laher , David Shupe , Ben Rusholme , Frank Masci , Carrie Nugent , "Rebbapragada, Umaa D (398J)" , Adam Miller , "Mansi M. Kasliwal" , Eric Bellm , Tom Barlow , Vandana Desai , Jason Surace , David Cook , Thomas A Prince , Yi Mei , Richard Dekany , Lin Yan Cc: Steve Groom , Lisa Storrie-Lombardi Minutes (6 / 7 / 2016) Present: Adam, Dave C., Ben, Eric, Tom B., Russ, Jason, Frank * PTF release preparation status: - new DB load files for the objects and associations table re-ingested by IRSA. This fixes the non-unique ObjectID issue. - all new reference images finalized (9,638 of them) and ingested by IRSA. - Eric to review draft of object-table column definitions forwarded by Frank; will eventually appear on user query interface. - Frank to forward Eric and Jason list of pids (and refids) that will be included in Fall release. Jason offering to remake sky-coverage maps of release contents. * New astrometric-calibration code was delivered by Dave S. for iPTF operations. Deployed and tested in realtime pipeline only by Russ. Cannot deploy into frameproc (nightly) pipeline yet due to an issue with PV-to-SIP distortion conversion when modifying CD matrix. Want to maintain consistency with archived image headers. Dave S. to explore when back from vacation. * New point-source realbogus for remainder of iPTF: on hold until Umaa has more time to explore. * Photometric calibration in iPTF operations now broken because of a recent change in the SDSS server URL when accessing DR12 products. Dave C. mentioned this and is offering to help resolve the issue. Thanks Dave. * Russ almost completed calibration (image stacking) pipelines for ZTF, primarily for generating bias and flat-field calibration products * Frank has first version of instrumental-calibration (science) pipeline working for ZTF. Currently missing astrometric calibration step, but will be included when Dave S. returns from vacation. Frank also started working on reference-image pipeline. * Frank forwarded Brad Cenko some specs to reimplement Eran’s ZOGY in Python, for insertion into ZTF realtime pipeline. * Halpha reprocessing: Dave C. testing Dave S.’s new astrometric calibration code. Frank to send Dave C. details of offline code and workaround to enable SExtractor to run on images with new WCS headers. * Ben requesting access to servers behind firewall at P48; submitted help desk ticket. Will ping Patrick Shopbell and others in person. Regards, Frank Hi All, Today 1PM, KS-300. Agenda: 1. PTF/iPTF release preparation status: lightcurve DB; new reference images; quality checks; coverage maps; object table column definitions. 2. New astrometric-calibration code for iPTF. 3. Status of new point-source realbogus for remainder of iPTF. 4. Halpha reprocessing. 5. ZTF pipeline development status. --- Goto Meeting: 1. Please join my meeting. https://global.gotomeeting.com/join/595527485 2. Use your microphone and speakers (VoIP) - a headset is recommended. Or, call in using your telephone. Dial +1 (213) 493-0622 Access Code: 595-527-485 Audio PIN: Shown after joining the meeting Meeting ID: 595-527-485 Regards, Frank From: Frank Masci Subject: Re: iPTF/ZTF Meeting Today Date: May 31, 2016 at 3:09:33 PM PDT To: Russ Laher , David Shupe , Ben Rusholme , Frank Masci , Carrie Nugent , "Rebbapragada, Umaa D (398J)" , Adam Miller , "Mansi M. Kasliwal" , Eric Bellm , Tom Barlow , Vandana Desai , Jason Surace , David Cook , Thomas A Prince , Yi Mei , Richard Dekany , Lin Yan Cc: Steve Groom , Lisa Storrie-Lombardi Minutes (5 / 31 / 2016) Present: Dave S., Russ, Jason, Frank * PTF release preparation status: - new DB load files for the objects and associations tables were redelivered to IRSA. This fixes the non-unique OID issue. - all new reference images finalized (9,638 of them) and delivered to IRSA. Three missing references recovered (Russ). - frank delivered a draft of the data-dictionary for the objects table (containing column definitions) to Eric for review. - frank requesting help on generating sky-coverage maps showing content in Fall release (Jason volunteered). * New point-source realbogus for remainder of iPTF: Umaa called to another project this week; says will deliver new version next week. IPAC will then deploy and test. * Dave S. has new astrometric-calibration code working for PTF. Hope is to get this integrated and tested in PTF pipeline(s) before revisiting the galactic plane in mid June. Russ suggested testing on a few epochs in real-time pipeline first, then frameproc. * Russ working on calibration (image stacking) pipelines for ZTF. Put together a flowchart of the processing steps for bias-image pipeline. Most steps carry over to the flat-generation pipeline. * Frank has first version of instrumental-calibration (science) pipeline working for ZTF. Currently missing astrometric calibration step, but will be included when Dave S. returns from vacation. * Halpha reprocessing: Dave S. tested new PTF astrometric calibration code on some Halpha images. Works nicely! * Completion of p15 (realtime pipeline) testing in ZTF environment: encountered a Python dependency issue related to incompatibility of RB model with version of scikit-learn (v0.17) on ZTF machines. Will revisit later when real-time pipeline is reimplemented. Regards, Frank On May 31, 2016, at 8:53 AM, Frank Masci wrote: Hi All, Today 1PM, KS-300. Agenda: 1. PTF/iPTF release preparation status: lightcurve DB; new reference images; quality checks; coverage maps; object table column definitions. 2. New astrometric-calibration code for iPTF. 3. Status of new point-source realbogus for remainder of iPTF. 4. Halpha reprocessing. 5. p15 testing in ZTF environment. 6. ZTF pipeline development status. --- Goto Meeting: 1. Please join my meeting. https://global.gotomeeting.com/join/595527485 2. Use your microphone and speakers (VoIP) - a headset is recommended. Or, call in using your telephone. Dial +1 (213) 493-0622 Access Code: 595-527-485 Audio PIN: Shown after joining the meeting Meeting ID: 595-527-485 Regards, Frank From: Frank Masci Subject: Re: iPTF/ZTF Meeting Today Date: May 24, 2016 at 3:25:33 PM PDT To: Russ Laher , David Shupe , Ben Rusholme , Frank Masci , Carrie Nugent , "Rebbapragada, Umaa D (398J)" , Adam Miller , "Mansi M. Kasliwal" , Eric Bellm , Tom Barlow , Vandana Desai , Jason Surace , David Cook , Thomas A Prince , Yi Mei , Richard Dekany , Lin Yan Cc: Steve Groom , Lisa Storrie-Lombardi Minutes (5 / 24 / 2016) Present: Dave S., Tom P., Russ, Ben, Adam, Jason, Frank * PTF release preparation status: - IRSA discovered multiple identical object identifiers in the objects table. Russ is regenerating the DB load files for the objects and associations that fixes this issue. There is also a checker in place to ensure that maximum ID is not exceeded. The new load files should be ready by end of this week. We will then perform some checks. - 9,638 new reference images were delivered last week. Wei found three cases(?) with missing metadata. These same ones don’t have PSF-fit photometry files. Russ to look into. - need data-dictionary for objects table that defines all the metrics (Frank). * New point-source realbogus for remainder of iPTF: Umaa says delivery should happen next week. IPAC will then deploy and test. * Halpha reprocessing: Dave S. to test new astrometric calibration code on some Halpha images provided by Dave C. * Completion of p15 (realtime pipeline) testing in ZTF environment: Ben and Russ working on it. * Bandwidth/throughput testing: “ztfx” server set up at IPAC. Ben working with Patrick to get access to server ports at P48. * ZTF pipeline development status: Frank working on instrumental/photometric calibration routines; Russ working on calibration-generation modules, including QA; Dave made good progress on astrometric-calibration (testing on PTF data looks superb). * Dave S: hoping to get new astrometric-calibration code integrated into PTF pipeline(s) before he leaves for vacation on June 2. Goal is to have this in place and tested before PTF revisits the galactic plane in mid June. Frank also requested a standalone script that executes the new astrometric-calibration code for use in the ZTF pipeline. * Adam: presented some new plots on his Star/Galaxy classifier. Goal is to have a more complete SDSS catalog in the Transients DB soon. This new table will have ~200million rows with perhaps 4 columns. Frank to ensure this table will carry forward to ZTF. * Jason: request for survey fields to simulate in next round of simulations. An example input schedule file was forwarded to Eric and Frank. There was also a request to simulate of order 250 new flats and bias exposures. Regards, Frank On May 24, 2016, at 8:43 AM, Frank Masci wrote: Hi All, Today 1PM, KS-300. Agenda: 1. PTF/iPTF release preparation status: lightcurve DB; new reference images; quality checks; coverage maps. 2. Status of new point-source realbogus for remainder of iPTF. 3. Halpha reprocessing. 4. p15 testing in ZTF environment. 5. Bandwidth/throughput testing update (P48 - IPAC). 6. ZTF pipeline development status. --- Goto Meeting: 1. Please join my meeting. https://global.gotomeeting.com/join/595527485 2. Use your microphone and speakers (VoIP) - a headset is recommended. Or, call in using your telephone. Dial +1 (213) 493-0622 Access Code: 595-527-485 Audio PIN: Shown after joining the meeting Meeting ID: 595-527-485 Regards, Frank From: Frank Masci Subject: Re: iPTF/ZTF Meeting Today Date: May 10, 2016 at 3:51:12 PM PDT To: Russ Laher , David Shupe , Ben Rusholme , Frank Masci , Carrie Nugent , "Rebbapragada, Umaa D (398J)" , Adam Miller , "Mansi M. Kasliwal" , Eric Bellm , Tom Barlow , Vandana Desai , Jason Surace , David Cook , Thomas A Prince , Yi Mei , Richard Dekany , Lin Yan Cc: Steve Groom , Lisa Storrie-Lombardi Minutes (5 / 10 / 2016) Present: Dave S., Russ, Ben, Eric, Adam, Tom B., Frank * PTF release preparation status and minor issues: - IRSA has ingested lightcurve DB load files; tables can be individually queried within Gator on irsatest. - Four PIDs containing anomalous RA position values will be dropped from the lightcurve tables. Corresponding image and catalog files will not for now. - Objects table has Nobs=‘null’ values. Origin yet unknown. Dave S. to explore. - Objects table data dictionary needs definitions for all lightcurve metrics (Frank to provide with Dave’s help). - 9638 new reference images delivered to irsa-inbox (Russ); currently being ingested by Wei. - another 462 reference images that need replacing are pending (Russ). - request for plots and documentation: e.g., coverage maps of epochal images and reference images in g and R (Frank and Jason). * New point-source realbogus model for new PTFIDE to be deployed into iPTF operations within the next couple of weeks. * Halpha reprocessing status: Dave C. executed Dave S.’s WCS-correction code; >90% of the images got fixed. Remainder pending analysis. * ZTF simulations: Jason about to redeliver new versions with small fixes. * Bandwidth/throughput testing P48 <---> IPAC: Ben performed testing using simulated ZTF images; took ~ 165sec to pull 16 compressed CCD image files using 4 rsync streams in parallel. Goal is 45 sec. Pushing data to P48 was faster (~45sec for 16 CCDs). This asymmetry needs to be understood. Testing continues. * ZTF instrumental calibration pipeline in progress (Frank) * Dave S. working on an improved astrometric-calibration method for iPTF; sounds promising for ZTF. Goal is to have something implemented/deployed by mid June when Galactic Plane observations recommence. * Russ has version 0 of ZTF ingest pipeline working; removed dependencies on manifest summary files; recomputing checksums for ingested data and comparing to header values as sanity check. Regards, Frank On May 10, 2016, at 9:14 AM, Frank Masci wrote: Hi All, Today 1PM, KS-300. Agenda: 1. PTF/iPTF release preparation status: lightcurve DB; new reference images; quality checks; coverage maps. 2. New point-source realbogus for remainder of iPTF (Umaa - in about 1 to 2 weeks) 3. Halpha reprocessing. 4. ZTF simulation status. 5. Testing in ZTF environment. 6. Bandwidth/throughput testing (P48 - IPAC). 7. ZTF pipeline development status; astrometric and photometric calibration. --- Goto Meeting: 1. Please join my meeting. https://global.gotomeeting.com/join/595527485 2. Use your microphone and speakers (VoIP) - a headset is recommended. Or, call in using your telephone. Dial +1 (213) 493-0622 Access Code: 595-527-485 Audio PIN: Shown after joining the meeting Meeting ID: 595-527-485 Regards, Frank From: Frank Masci Subject: Re: iPTF/ZTF Meeting Today Date: April 26, 2016 at 4:29:41 PM PDT To: Russ Laher , David Shupe , Ben Rusholme , Frank Masci , Carrie Nugent , "Rebbapragada, Umaa D (398J)" , Adam Miller , "Mansi M. Kasliwal" , Eric Bellm , Tom Barlow , Vandana Desai , Jason Surace , David Cook , Thomas A Prince , Yi Mei , Richard Dekany , Lin Yan Cc: Steve Groom , Lisa Storrie-Lombardi Minutes (4 / 26 / 2016) Present: Dave S., Dave C., Yi M., Russ, Jason, Ben, Eric, Lin, Adam, Frank * Discussed content for PTF/iPTF Fall release: lightcurve DB; epochal images and catalog files; additional reference images. Date range of epochal-based products is 3/1/2009 - 1/28/2015. Frank to ensure that: (i) only epochal product files with proc image ids (pIds) spanning the lightcurve DB content be released; (ii) any pids withheld from previous public releases (<= 12/31/2012) but that are now in the LC timespan be released; (iii) all reference image products made exclusively from epochal data acquired > 1/28/2015 to be excluded from release. * Related to previous bullet: request for coverage maps for new products in Fall release. Also total volume, source counts. * Regeneration of DB load files to support lightcurve release is in progress, namely expanding numerical values in scientific notation and replacing nulls with blanks. Should be complete by tomorrow (4/27) -- Russ. * ZPVM-stability experiment by reprocessing 4 nights of PTF data: in progress (to support ZTF flat-fielding decision). * Halpha reprocessing: Dave C. used Dave S.’s WCS-correction code. All good. Dave C. requested disk space for local pipeline runs. Also request by IPAC team to keep formatting of final (archival) products the same. Also to generate in a manner that makes products easy to archive later. Lin happy to help with vetting of products before archival. * Adam requested booking KS 410 for iPTF Summer School to help with tutorial sessions (Frank). * Lin requested that everyone register for the Summer School and pay the fee, including speakers. Frank to talk to Lisa about charge accounts. * Ben now has the P48 account to use for throughput testing. Has also working on testing of PTF pipelines in ZTF environment * Frank has early phase of ZTF pipeline implemented: decompression, FITS extension splitting, floating-bias correction. * Adam mentioned that a special two-filter (R, g), same night, same field experiment will be carried out on P48. Regards, Frank On Apr 26, 2016, at 8:44 AM, Frank Masci wrote: Hi All, Today 1PM, KS-300. Agenda: 1. PTF/iPTF public release content, scope, cutoff dates, embargoed data(?), quality checks. 2. Load status for lightcurve DB. 3. P48 account(s) for throughput/transfer testing. 4. Testing of PTF pipelines in ZTF environment. 5. status of reprocessing to support ZPVM stability experiment. 6. ZTF pipeline development status. 7. Halpha reprocessing status. --- Goto Meeting: 1. Please join my meeting. https://global.gotomeeting.com/join/595527485 2. Use your microphone and speakers (VoIP) - a headset is recommended. Or, call in using your telephone. Dial +1 (213) 493-0622 Access Code: 595-527-485 Audio PIN: Shown after joining the meeting Meeting ID: 595-527-485 Regards, Frank From: Frank Masci Subject: Re: iPTF/ZTF Meeting Today Date: April 19, 2016 at 3:02:11 PM PDT To: Russ Laher , David Shupe , Ben Rusholme , Frank Masci , Carrie Nugent , "Rebbapragada, Umaa D (398J)" , Adam Miller , "Mansi M. Kasliwal" , Eric Bellm , Tom Barlow , Vandana Desai , Jason Surace , David Cook , Thomas A Prince , Yi Mei , Richard Dekany Cc: Steve Groom , Lisa Storrie-Lombardi Minutes (4 / 19 / 2016) Present: Dave S., Adam, Eric, Jason, Ben, Russ, Tom P., Tom B., Frank * Status of preparations for PTF/iPTF lightcurve release: big sources table has been ingested; indices are currently being built; will then proceed to load “objects”, “associations”, and “epochs” tables. Expect loading to be complete by ~mid May. Simple query interface using Gator and table-view tools is envisaged. * Discussion on lightcurve DB content/usage for ZTF: expect decision on how to proceed following Maryland meeting (~late May). * Related to previous bullet: also discussed some details of source matching to support lightcurve generation for ZTF. Dave S. mentioned probably don’t need to create intermediate “nightly sources" HDF files. Only need a way to store timestamp of last catalog used to create source-match file. Eric: a match-file of some sort needs to persist in operations (per readout-channel) to enable ongoing relative photometry and computation of LC metrics. Format is TBD. * Frank mentioned new storage estimates for keeping the bare minimum of difference image products for ZTF: ~ 2PB for entire 3 yr survey (compressed). Frank to check on cost. Tom P. mentioned we can prioritize on regions of sky to store; galactic-plane is one of them. * ZTF image simulation status: new set was generated by Jason according to latest camera-packaging specs. These can be used for pipeline development. Future: need to fold in a more realistic survey plan. No hurry on this. Another detail is that the simulator currently uses Vizier to grab SDSS sources to overlay. Future: need a more efficient way to grab these. * ZPVM-stability experiment by reprocessing 4 nights of PTF data: in progress. Russ: waiting for bad weather(!) * iPTF-Gaia collaborative program to start during early June. Frank to keep track of pid’s to omit from public release. Regards, Frank On Apr 19, 2016, at 9:19 AM, Frank Masci wrote: Hi All, Today 1PM, KS-300. Agenda: 1. PTF/iPTF lightcurve release preparation 2. ZTF source-matching/relative photometry plans. Feeds into ZTF LC database use cases. 3. Archival of subtraction images for ZTF; new estimates: ~2.5 TB/night or ~2 PB/3years compressed. 4. ZTF simulation status 5. ZPVM stability (low-frequency flat) experiment using PTF 6. iPTF-Gaia program 7. Halpha reprocessing status 8. Not this meeting, but in progress: ops and transients DB schema redesign; data-link bandwidth testing; filesystem for ztf processing; test of ptf pipelines in ztf environment; catalogs to support astrometric & phot. calibration. --- Goto Meeting: 1. Please join my meeting. https://global.gotomeeting.com/join/595527485 2. Use your microphone and speakers (VoIP) - a headset is recommended. Or, call in using your telephone. Dial +1 (213) 493-0622 Access Code: 595-527-485 Audio PIN: Shown after joining the meeting Meeting ID: 595-527-485 Regards, Frank From: Frank Masci Subject: Re: iPTF/ZTF Meeting Today Date: April 12, 2016 at 3:23:10 PM PDT To: Russ Laher , David Shupe , Ben Rusholme , Frank Masci , Carrie Nugent , "Rebbapragada, Umaa D (398J)" , Adam Miller , "Mansi M. Kasliwal" , Eric Bellm , Tom Barlow , Vandana Desai , Jason Surace , David Cook , Thomas A Prince , Yi Mei , Richard Dekany Cc: Steve Groom , Lisa Storrie-Lombardi Minutes (4 / 12 / 2016) Present: Dave S., Yi M., Jason, Tom P., Russ, Ben, Adam, Tom B., Frank * Lightcurve release prep status: some instances of null-valued columns were uncovered in the “Source” table by IRSA which we thought were not null in the deliveries. An example is the mag_auto measure which is used for lightcurve generation. We have examples that violated this non-null constraint to explore further. * ongoing source matching: Py 2.7 code was delivered to ops. To do: load new relative photometry results into ops DB (Dave S.). Plan on another run of source-matching at end of April/early May. * New point-source RealBogus: Umaa contacted Frank after today’s meeting and said a new version should be ready by early May. * ZTF sim status: new round of sims were generated (Jason); some details need to be folded in, one is related to next bullet. * Frank to send a revised version of WCS keyword specs to the camera team. Dave S. says it’s best to use the CD matrix convention and not PC + CDELTs. Frank then to give go-ahead on a new simulation run. * zpvm-stability experiment: Eric to select several photometric nights to reprocess; Jason mentioned it’s best to avoid data earlier than Oct 2010 due to the fogging problem; Frank to pick suitable static uber-flat to use for all runs; Russ to reprocess. Previous products should still be retrievable. * Halpha reprocessing: Dave S. forwarded Dave C. astrometric correction code with documentation. * Tom B: upcoming schedule has another round of Halpha calibrations; will let Russ know if there are any new field IDs. * Tom P: mentioned collaborative effort with Gaia to use P48 to help fill in temporal sampling. Requested that such data (images, catalogs and lightcurves therefrom) be withheld from public, but can be released to iPTF collaboration only. * Adam: mentioned a hack was implemented to forward metadata on interesting candidates from treasures page to LBL so proper names can be assigned. * Dave S: optimizing SCAMP (astrometric calibration) for PTF: a version of “reSCAMP”, but also a version that can read in prior distortion coefficients with OBJRA, OBJDEC as proxies for CRVAL1,2 for updating later; also reported on LSST discussions regarding WCS libraries and distortion conventions. * Russ: working on ZTF Operations Database Schema and more. * Ben: still waiting for a P48 account from Patrick Shopbell so can start latency/bandwidth testing; also built IDL on ZTF machines. Regards, Frank On Apr 12, 2016, at 9:17 AM, Frank Masci wrote: Hi All, Today 1PM, KS-300. Agenda: 1. Lightcurve release preparation status 2. Ongoing source-matching/relative photometry status 3. New point-source RealBogus model for iPTF 4. ZTF simulation status 5. zpvm-stability experiment: request to reprocess several photometric nights of PTF data using a fixed uber-flat. 6. Propagation of metadata on interesting candidates from treasures scanning page to LBL. 7. Halpha reprocessing status: astrometric correction and recalibration codes 8. Not this meeting, but in progress: ops and transients DB schema redesign; data-link bandwidth testing; filesystem partitioning for ztf processing; test of ptf pipelines in ztf environment; test catalogs to support astrometric & phot. calibration. --- Goto Meeting: 1. Please join my meeting. https://global.gotomeeting.com/join/595527485 2. Use your microphone and speakers (VoIP) - a headset is recommended. Or, call in using your telephone. Dial +1 (213) 493-0622 Access Code: 595-527-485 Audio PIN: Shown after joining the meeting Meeting ID: 595-527-485 Regards, Frank From: Frank Masci Subject: Re: iPTF/ZTF Meeting Today Date: April 5, 2016 at 3:36:12 PM PDT To: Russ Laher , David Shupe , Ben Rusholme , Frank Masci , Carrie Nugent , "Rebbapragada, Umaa D (398J)" , Adam Miller , "Mansi M. Kasliwal" , Eric Bellm , Tom Barlow , Lin Yan , physarah@gmail.com, Vandana Desai , Jason Surace , David Cook , Thomas A Prince , Yi Mei , Richard Dekany Cc: Steve Groom , Lisa Storrie-Lombardi Minutes (4 / 5 / 2016) Present: Dave S., Yi M., Dave C., Jason, Ben, Eric, Adam, Frank * Lightcurve preparation status: all DB load-files to support Fall release have been delivered to IRSA. Integration underway; planning on user-interface is also underway. * Dave S. planning to update operations database with relative photometry corrections; also to consolidate latest source-matching/rel-phot code and deliver to PTF repository, possibly under a Python 3 branch. * The IPAC/PTF Image Differencing Engine (PTFIDE) discovered a new Type Ia supernova through Adam Miller’s diligent scanning! * Chip 11 now appears to be alive(!). Realtime pipeline still generates products from this chip but Frank is unsure if this is still processed though the photometric pipeline. Will check with Russ. * Feedback to be provided to Camera team on metadata and packaging specs for raw image data. Frank to collect suggestions very soon for review. * Following receipt of updated packaging/metadata specs, Frank will forward to Jason so ZTF simulator can be updated. * Halpha processing: Dave S. to forward Dave C. astrometric correction code and recipe. * Adam M. (through Yi Cao) requesting that a new script be written to propagate metadata on interesting candidates found on IPAC scanning page to the LBL database. This will enable proper naming and marshaling. Regards, Frank On Apr 5, 2016, at 8:39 AM, Frank Masci wrote: Hi All, Today 1PM, KS-300. Agenda: 1. Lightcurve release preparation status 2. Relative photometry updates in ops(?) 3. New point-source RealBogus model for iPTF 4. ztf raw-image packaging and metadata in review: to feed into new round of ZTF simulations soon 5. Halpha reprocessing status: astrometric correction and recalibration codes 6. Not this meeting, but in progress: ops and transients DB schema redesign; data-link bandwidth testing; filesystem partitioning for ztf processing; design of ztf data-processing flow; test of ptf pipelines in ztf environment; test catalogs to support astrometric & phot. calibration. --- Goto Meeting: 1. Please join my meeting. https://global.gotomeeting.com/join/595527485 2. Use your microphone and speakers (VoIP) - a headset is recommended. Or, call in using your telephone. Dial +1 (213) 493-0622 Access Code: 595-527-485 Audio PIN: Shown after joining the meeting Meeting ID: 595-527-485 Regards, Frank From: Frank Masci Subject: Re: iPTF/ZTF Meeting Today Date: March 29, 2016 at 4:42:14 PM PDT To: Russ Laher , David Shupe , Ben Rusholme , Frank Masci , Carrie Nugent , "Rebbapragada, Umaa D (398J)" , Adam Miller , "Mansi M. Kasliwal" , Eric Bellm , Tom Barlow , Lin Yan , physarah@gmail.com, Vandana Desai , Jason Surace , David Cook , Thomas A Prince , Yi Mei , Richard Dekany Cc: Steve Groom , Lisa Storrie-Lombardi Minutes (3 / 29 / 2016) Present: Dave S., Yi M., Jason, Ben, Eric, Russ, Adam, Frank * all lightcurve DB load-files for the “Objects”, “Associations”, and “Epochs” table are complete (thanks Russ!), and reside in /stage/ptf-irsainbox/loadFilesForLightcurveDB/. * as a spot-check, a script to count the number of records (rows) in the above DB load-files is currently running. * matchfiles are now up to date to March 16, 2016 in operations (thanks Dave!). Relative photometry not yet updated in ops DB and there’s no strong need to do this soon. * new matchfile code to be delivered to operations (Dave). * R-band reals to support training of new PTFIDE subtractions is done. This is now in Umaa’s hands at JPL. * simulation status: Jason delivered 80 simulated raw compressed CCD images. These are sufficient for throughput testing. We are waiting on Roger to provide details on camera geometry and layout of CCDs/overscan regions therein. Metadata specs for image headers were received this morning. * request for simulated bias images, packaged/compressed in same format as science-survey files (Jason to explore). * Halpha reprocessing status: Dave S. documented his astrometric correction code for Dave C. to run. * design/hashing of ZTF data-processing flow (Frank). Regards, Frank On Mar 29, 2016, at 8:55 AM, Frank Masci wrote: Hi All, Today 1PM, KS-300. Agenda: 1. Lightcurve release preparation status - DB load-file generation for “Objects”, “Associations”, and “Epochs” table - documentation - catch-up of matchfile generation with relative phot; status of code? 2. Reprocessing of “reals” for retraining RealBogus to support new PTFIDE done; training in progress 3. Status of ZTF simulations; raw-image metadata specs received; waiting for geometric info from camera team 4. Halpha reprocessing status: astrometric correction and recalibration codes 5. New streak classifier ready for integration in realtime pipeline [discussion with Umaa to proceed] 6. Not this meeting, but in progress: hardware purchases; ops and transients DB schema redesign; data-link bandwidth testing; software building/config on ztf machines; design of ztf data-processing flow. --- Goto Meeting: 1. Please join my meeting. https://global.gotomeeting.com/join/595527485 2. Use your microphone and speakers (VoIP) - a headset is recommended. Or, call in using your telephone. Dial +1 (213) 493-0622 Access Code: 595-527-485 Audio PIN: Shown after joining the meeting Meeting ID: 595-527-485 Regards, Frank From: Frank Masci Subject: Re: iPTF/ZTF Meeting Today Date: March 22, 2016 at 2:51:34 PM PDT To: Russ Laher , David Shupe , Ben Rusholme , Frank Masci , Carrie Nugent , "Rebbapragada, Umaa D (398J)" , Adam Miller , "Mansi M. Kasliwal" , Eric Bellm , Tom Barlow , Lin Yan , physarah@gmail.com, Vandana Desai , Jason Surace , David Cook , Thomas A Prince , Yi Mei , Richard Dekany Cc: Steve Groom , Lisa Storrie-Lombardi Minutes (3 / 22 / 2016) Present: Dave S., Dave C., Yi M., Eric, Russ, Jason, Adam, Frank * matchfile set for lightcurve release was refrozen following a small fix to omit uncalibrated photometry prior to relative photometry step being run. Frozen copy is now consistent with operations set. The latter will be updated to support ongoing source matching. * sample DB load-files for “Objects”, “Associations”, and “Epochs” table containing ~2000 objects will be generated soon and forwarded to IRSA for review/testing. * related to DB load-file generation: any lightcurve with ngoodobs < 4 will have its "non-best” metric columns set to ‘null' while lightcurves with nbestobs < 4 will have its “best-prefixed” metric columns set to ‘null’. * status of reprocessing of R-band reals to support retraining of new PTFIDE subtractions: should be done any time now. In the interest of time, there was also a brief discussion on whether we should retrain using only R-band and use the model therefrom on g-band data (TBD). * analysis of impact of image-compression on co-addition: conclusion: increase in overall noise-sigma due to quantization error is ~0.3% (or 3 millimag), independent of number of images co-added. * simulation status: waiting on Roger to provide details on camera geometry, in particular orientation of each CCD and overscan regions. Jason to forward Frank latest set of simulation images to test. * Halpha reprocessing status: Dave S. to forward Dave C. the astrometric-correction code in a couple of weeks from now so Dave C. can run. Also, a discussion on how Dave C. can retrieve image cutouts from IRSA using API scripts. Examples on how to do this are in a set of slides on the PTF wiki. Regards, Frank On Mar 22, 2016, at 9:36 AM, Frank Masci wrote: Hi All, Today 1PM, KS-300. Agenda: 1. Lightcurve release preparation status - sample DB load-file generation for “Objects”, “Associations”, and “Epochs” table (Null’ing plan?) - code deliveries to operations to support ongoing matchfile generation, with relative phot. 2. Reprocessing of “reals” for retraining RealBogus to support new PTFIDE (expected completion ~ 1.5 days) 3. Ingestion of new star-galaxy classification tables into the PTF Transients DB 4. Impact of image compression on co-addition (document updated): http://web.ipac.caltech.edu/staff/fmasci/home/miscscience/compression_test_ptf.pdf 5. Status of ZTF simulations [waiting for details from Camera Team] 6. Halpha reprocessing status 7. Not this meeting, but in progress: hardware purchase planning; DB schema redesign; data-link bandwidth testing; software building/config on ztf machines. --- Goto Meeting: 1. Please join my meeting. https://global.gotomeeting.com/join/595527485 2. Use your microphone and speakers (VoIP) - a headset is recommended. Or, call in using your telephone. Dial +1 (213) 493-0622 Access Code: 595-527-485 Audio PIN: Shown after joining the meeting Meeting ID: 595-527-485 Regards, Frank From: Frank Masci Subject: Re: iPTF/ZTF Meeting Today Date: March 15, 2016 at 4:49:29 PM PDT To: Russ Laher , David Shupe , Ben Rusholme , Frank Masci , Carrie Nugent , "Rebbapragada, Umaa D (398J)" , Adam Miller , "Mansi M. Kasliwal" , Eric Bellm , Tom Barlow , Lin Yan , physarah@gmail.com, Vandana Desai , Jason Surace , David Cook , Thomas A Prince , Yi Mei , Richard Dekany Cc: Steve Groom , Lisa Storrie-Lombardi Minutes (3 / 15 / 2016) Present: Dave S., Dave C., Yi M., Ben, Eric, Russ, Jason, Tom B., Frank * Frozen matchfile set for lightcurve release currently being vetted by team at Cahill (Eric). Some epochs with NaN’d absolute ZPs were reported (see next bullet). * Dave S. reported that ~0.4% of the pids (proc image IDs) covering the release interval (3/1/09 -- 1/28/15) were identified as having NaN’d absolute ZPs. These will be omitted during construction of the DB load-files for the “associations” and “epochs” tables. * Sample DB load-file generation for “objects”, “associations”, and “epochs” tables in progress (Russ); need to decide on what values to interpret as “null”. * Russ to forward Eric a sample of NaN and Inf valued LC metrics to check their origin and whether it makes sense to “null” these out. * Dave S. to deliver latest code updates to ops to enable ongoing matchfile generation with relative photometry. * Existing (old) matchfiles from operations are now backed up on sbx5. Operations versions are currently being replaced with the frozen set to support ongoing updates using the latest epochs. * When matchfiles in operations are updated and Dave’s new code delivered, all epochs will be rematched up to March 2016(?). * Status of 16,000+ new ref images and reprocessed versions for archive? Frank to check with Wei Mi. * Frank to start reprocessing of “real” transients to support retraining of RealBogus on new PTFIDE very soon. * Frank looked at compression strategy for ZTF and impact on noise/information retained following decompression. Looks promising. TBD: impact on image co-addition and expected sensitivity loss therefrom is not yet explored. * ZTF simulation effort in progress. Jason to provide a sample of 20 or so CCDs by end of March according to packaging and compression specs. * Status of importing new star/galaxy classification tables into Transients DB: Russ waiting for Adam’s reply on whether R > 22 sources should be excluded. * Request from Jason to install IDL on ZTF cluster for purpose of generating multiple nights of simulated ZTF data. Desire to have IDL installed by mid April (confirmed by Ben). * Halpha reprocessing: WCS fixes identified by Dave S, although some cases still in limbo. Key difference is that scamp was rerun on image data with pre-existing (better) astrometry than otherwise would have been used. Dave C. to test on four fields and then check with Russ for integration into reprocessing script that also applies photometric and ZPVM calibrations. * Brief discussion on what external astrometric and photometric reference catalogs to use for ZTF, or rather, to use as placeholders for pipeline development: goal is Gaia for astrometry and PS1 for photometry. For development and testing, there was mention of using UCAC4+SDSS for astrometry and SDSS overlap footprint for photometry, in a format that can be adapted for other (larger) catalogs in future. Regards, Frank On Mar 15, 2016, at 9:19 AM, Frank Masci wrote: Hi All, Today 1PM, KS-300. Agenda: 1. Lightcurve release preparation status - spotchecks on frozen matchfiles - backup of operations matchfiles (sbx3); copy frozen set into operations with DB updates - Generate sample DB load-files for “Objects”, “Associations”, and “Epochs” table - code deliveries to operations to support future PTF/iPTF match file generation (with relative phot) 2. Another 16k+ reference images to be archived soon (release to collaboration TBD) 3. Reprocessing of “reals” for retraining RealBogus to support new PTFIDE 4. Ingestion of new star-galaxy classification tables into the PTF Transients DB 5. Compression study on raw PTF camera image files (looks encouraging for ZTF): http://web.ipac.caltech.edu/staff/fmasci/home/miscscience/compression_test_ptf.pdf 6. Halpha reprocessing status 7. Not this meeting, but in progress: hardware purchase to support near-term development and testing; future hardware budgeting; DB schema redesign; ZTF camera image packaging specs; data-link bandwidth throughput testing. --- Goto Meeting: 1. Please join my meeting. https://global.gotomeeting.com/join/595527485 2. Use your microphone and speakers (VoIP) - a headset is recommended. Or, call in using your telephone. Dial +1 (213) 493-0622 Access Code: 595-527-485 Audio PIN: Shown after joining the meeting Meeting ID: 595-527-485 Regards, Frank From: Frank Masci Subject: Re: iPTF/ZTF Meeting Today Date: March 8, 2016 at 3:54:24 PM PST To: Russ Laher , David Shupe , Ben Rusholme , Frank Masci , Carrie Nugent , "Rebbapragada, Umaa D (398J)" , Adam Miller , "Mansi M. Kasliwal" , Eric Bellm , Tom Barlow , Lin Yan , physarah@gmail.com, Vandana Desai , Jason Surace , David Cook , Thomas A Prince , Yi Mei , Richard Dekany Cc: Steve Groom , Lisa Storrie-Lombardi Minutes (3 / 8 / 2016) Present: Dave S., Dave C., Ben, Eric, Russ, Adam, Tom B., Yi M., Frank * Lightcurve matchfiles are currently being copied to Marshal so team can take a closer look (Eric). * Dave S. to deliver latest code updates to ops to enable future matchfile generation with relative photometry. * Russ to backup existing matchfiles in operations then copy new versions (from sbx5) into operations (sbx3). The latest relative photometry code (from Dave S.) will then be run to update/populate database info and re-synch everything. * When updates made in ops, re-match all new epochal photometry acquired since Sep 2015 (end date for frozen matchfiles) with relative photometry included. Goal is to do this by ~ Mar 23 to support COO proposal deadline (Apr 1). * Russ to revisit IRSA-DB load-file generation (with suggestions from Dave S.). Goal is to get a small sample of objects (with epochal associations) for Steve to experiment with. The final-release pid list provided by Frank should be cross-checked while generating the load files. * Frank to ping Serge M. on new htm20 code to assist with the load-file generation. * Frank done with PTFIDE fixes for now (to morph into ZTFIDE later). Talked with Umaa from JPL earlier today on getting a new set of “real” and “bogus” samples to reprocess and support retraining of the RealBogus classifier. This should carry us through to end of iPTF. * Frank working on raw-image compression strategies for ZTF (in consultation with camera team). * Russ to import new star/galaxy tables from Adam into Transients DB (two tables need to be wiped and reloaded: “refimallsources” and “refim_r”, where latter stores the prob.star > 0.975 classifications, *I think*). * Tom B. reported on continued problems with the drive on P48. Also reported on upcoming schedule. * Dave S. reported on reprocessing of some Halpha image data to explore astrometry. Improvements were identified. Plan is to fold this into later reprocessing of Halpha data (in consultation with Dave C.). * Ben working on software builds on ztf cluster; ansible becoming a challenge; resorted to “bash’ing” the system; also to ping Patrick Shopbell (again). Regards, Frank On Mar 8, 2016, at 9:33 AM, Frank Masci wrote: Hi All, Today 1PM, KS-300. Agenda: 1. Lightcurve release preparation status - spot checks on frozen matchfiles - copy matchfiles to operations (sbx3); also update/insert entries in ops DB - IRSA DB load-file generation using frozen matchfiles - code deliveries to support future PTF/iPTF matchfile generation 2. Another 16k+ reference images to be archived soon (release to collaboration TBD) 3. PTFIDE updates done: plans to retrain RealBogus on short-order. 4. Ingestion of new star-galaxy classification tables into the PTF Transients DB. 5. Halpha reprocessing status 6. Not this meeting, but in progress: discussions with ZTF camera team on raw-image packaging and compression; estimates on data volumes, data rates, and processing runtimes to guide hardware purchases in near future; qa tables in ZTF ops DB redesigned and made leaner; data-link bandwidth throughput testing. --- Goto Meeting: 1. Please join my meeting. https://global.gotomeeting.com/join/595527485 2. Use your microphone and speakers (VoIP) - a headset is recommended. Or, call in using your telephone. Dial +1 (213) 493-0622 Access Code: 595-527-485 Audio PIN: Shown after joining the meeting Meeting ID: 595-527-485 From: Frank Masci Subject: Re: iPTF/ZTF Meeting Today Date: March 1, 2016 at 4:42:55 PM PST To: Russ Laher , David Shupe , Ben Rusholme , Frank Masci , Carrie Nugent , "Rebbapragada, Umaa D (398J)" , Adam Miller , "Mansi M. Kasliwal" , Eric Bellm , Tom Barlow , Lin Yan , physarah@gmail.com, Vandana Desai , Jason Surace , David Cook , Thomas A Prince , Yi Mei , Richard Dekany Cc: Steve Groom , Lisa Storrie-Lombardi Minutes (3 / 1 / 2016) Present: Dave S., Yi M., Jason, Eric, Russ, Adam, Frank * matchfile status to support lightcurve release: we now have a frozen version (thank you Dave!). These are now ready to be pulled apart to generate the DB load-files for IRSA. - Russ has started to code some of the scripts to enable the load-file generation. A small sample of these load-files will be generated soon for review. - Eric to copy new matchfiles over to galactic marshall for spot-checking. - Also discussed plans to copy the new matchfiles into operations system with relevant DB updates (no urgency on this). Russ to procure disk space if possible so old versions can coexist for some time. - Frank to check with Steve on (i) final list of nights/pids that will be bulk-loaded into “Sources” table to ensure consistency with sid’s in “associations” table (cut off date is 1/31/2015?); (ii) confirm that nights identified as “maybes” will not be loaded. - Frank to ensure that missing or non-computable metrics destined for the “Objects” table are appropriately “null’d". - Dave to complete code updates (under Python 3?) and deliver to ops to streamline matchfile generation for future PTF releases (e.g., all repair steps, NaN checks?, pId consistency checks, etc..). * brief discussion on compression of ZTF raw image data by camera software: Frank to explore optimal noise-bit truncation vs compression ratio (goal is ~ x3). Will need to consider consequences for science applications (e.g., deep co-adds). * discussion on how to better “lockdown” reference image versions in future: e.g., those initially used to seed source-matching for lightcurve generation vs new/revised versions for other applications. * Frank to provide advance warning to Jason before requesting simulated ZTF image data. * Dave S. checked the astrometry for a few Halpha fields provided by Dave C. Those that defaulted to using astrometry.net (via infobits) looked worse around edges, although sample is small. There’s a possibility of reprocessing affected images offline then using updated WCS solutions to replace old ra,dec positions in SExtractor catalogs using their native x,y positions. * Adam presented some performance plots of the new star-galaxy classifier. New versions of the star/galaxy tables are ready for ingestion into the Transients Database (at least into the “refimallsources" table). There’s also a “refim_r” table with a probability cut to retain only stars -- this needs to be discussed with the scanning team. Regards, Frank On Mar 1, 2016, at 9:22 AM, Frank Masci wrote: Hi All, Today 1PM, KS-300. Agenda: 1. Lightcurve release preparation status - 56,363 matchfiles now frozen under sbx5; includes relative photometry - spot checks needed - copy and replace files in operations (sbx3); also update/insert entries in ops DB - nights or pIDs to omit from release (IRSA not to load these into "Sources” table)? - strategy for IRSA DB load-file generation (generate htm20, unique objectIDs from matchIDs, 1-based counters..) - plans to support future PTF/iPTF match file (and load-file) generation? 2. Another 16k+ reference images to be archived soon 3. Updates to pipeline-flow plans in ZTF Software definitions document 4. (Z)PTFIDE updates (still in progress) 5. Not this meeting, but in progress: discussions with ZTF camera team on image packaging and format; estimates on data volumes, data rates and processing runtimes in review to guide hardware purchases in near future. 6. Halpha reprocessing status Regards, Frank --- Goto Meeting: 1. Please join my meeting. https://global.gotomeeting.com/join/595527485 2. Use your microphone and speakers (VoIP) - a headset is recommended. Or, call in using your telephone. Dial +1 (213) 493-0622 Access Code: 595-527-485 Audio PIN: Shown after joining the meeting Meeting ID: 595-527-485 From: Frank Masci Subject: Re: iPTF/ZTF Meeting Today Date: February 23, 2016 at 5:29:53 PM PST To: Russ Laher , David Shupe , Ben Rusholme , Frank Masci , Carrie Nugent , "Rebbapragada, Umaa D (398J)" , Adam Miller , "Mansi M. Kasliwal" , Eric Bellm , Tom Barlow , Lin Yan , physarah@gmail.com, Vandana Desai , Jason Surace , David Cook , Thomas A Prince , Yi Mei Cc: Steve Groom , Lisa Storrie-Lombardi Minutes (2 / 23 / 2016) Present: Eric, Dave S., Yi M., Jason, Adam, Russ, Dave C., Tom B., Frank, * Lightcurve preparation status: ~8000 match files were found to be missing; procedure to regenerate is straightforward and a frozen set is expected to be in place by mid next week. Relative photometry will then run. Copies in ops (with DB pointers) will then be updated. We will then focus on writing software to generate the IRSA DB load files. Schema for lightcurve DB is still in review. * Discussion on strategies to trim extracted sources for constructing a “usable" lightcurve DB for ZTF. E.g., cuts in s/n; removal of extended sources…TBD. Better understanding of survey strategy needed. * Frank to send Eric FITS images of the 3pi source-density maps. * Halpha reprocessing status: Dave C. reported on some analysis of the astrometric calibration with a possibility of reprocessing four fields of interest. He will summarize his finds soon. * Shallower references in g-band currently in progress (Russ). * Frank to check with Steve on whether they’re ready to ingest and archive another ~10k reference images. * Following the meeting there was a brief discussion on buying a MATLAB license for ZTF. All is still TBD. Regards, Frank On Feb 23, 2016, at 8:56 AM, Frank Masci wrote: Hi All, Today 1PM, KS-300. Agenda: 1. Lightcurve release preparation - repair of corrupted match files under sbx5; - run relative photometry to compute/apply ZP corrections to all match files; - cross-check against backup copies (that didn’t have dZP applied); - verify specs on metadata to load into IRSA DB. - write software to pull apart hdf5 files and generate the various IRSA DB load-files. 2. New draft of lightcurve schema 3. 3pi source density maps (from aperture & PSF-fit catalogs) 4. New reference image products to archive; anything else? 5. Summary of feedback on ZTF Software definitions document 6. PTFIDE updates (on hold) 7. Halpha reprocessing status Regards, Frank From: Frank Masci Subject: Re: iPTF/ZTF Meeting Today Date: February 16, 2016 at 5:13:45 PM PST To: Russ Laher , David Shupe , Ben Rusholme , Frank Masci , Carrie Nugent , "Rebbapragada, Umaa D (398J)" , Adam Miller , "Mansi M. Kasliwal" , Eric Bellm , Tom Barlow , Lin Yan , physarah@gmail.com, Vandana Desai , Jason Surace , David Cook , Thomas A Prince , Yi Mei Cc: Steve Groom , Lisa Storrie-Lombardi Minutes (2 / 16 / 2016) Present: Ben, Dave S., Eric, Russ, Dave C., Yi M., Frank - match file corruption issues identified and fixes expected to be complete by end of this week: <~ 5% of the match files have duplicate entries in their source tables and were confusing the relative photometry step. Also plan to check number of instances with missing absolute calibration (NaN’d photometry). - draft schema for lightcurve database was presented (Frank). Need to sit down with Steve to iron out details, in particular data-access control. Brief discussion on which metrics to include from match-files (TBD). - last night’s image cutouts from realtime pipeline not accessible through CGI. Was this related to power glitch over weekend? Perhaps not all disks are being mounted. To be explored. - touched on s/w definitions and deliverables document. More feedback received and is currently being digested. - Halpha reprocessing: Dave C. presented example images with bad astrometry. Task is to check on frequency of bad cases and whether science can still proceed with the good ones. Also possibility of reprocessing some of the failures using later software/config files since failures are from long ago that may have used older astrometric reference catalogs (TBD). - PTF codebase progressively being built/installed on ztfops machines for testing. Regards, Frank On Feb 16, 2016, at 9:48 AM, Frank Masci wrote: Hi All, Today 1PM, KS-300. Agenda: 1. Lightcurve release preparation and status. Short-term plan: - repair corrupted match files in ops (need resolution by Feb 15); - backup all updated match files to either ptfdepot, sbx5 or Cahill; - run relative photometry to compute/apply ZP corrections to match files in ops; - cross-check against backup copies (that didn’t have dZP applied); - remove backup copy; - freeze ZP-corrected (hdf5) match files intended for release by copying to separate partition; - revisit schema for IRSA DB loading and confirm load-file specs with Steve; - draft specs to support loading of “object”, “associations”, “dZP” tables for IRSA; - analyses and documentation: validation of ZP corrections (rms checks); - write software to pull apart hdf5 files and generate the various IRSA load-files. 2. Draft of lightcurve schema to support IRSA DB loading (load-files) 3. ZTF Software, Deliverables & Plans document: ready for distribution. 4. PTFIDE updates (on hold) 5. Halpha reprocessing status. Regards, Frank --- Goto Meeting: 1. Please join my meeting. https://global.gotomeeting.com/join/595527485 2. Use your microphone and speakers (VoIP) - a headset is recommended. Or, call in using your telephone. Dial +1 (213) 493-0622 Access Code: 595-527-485 Audio PIN: Shown after joining the meeting Meeting ID: 595-527-485 From: Frank Masci Subject: Re: iPTF/ZTF Meeting Today Date: February 9, 2016 at 3:29:32 PM PST To: Russ Laher , David Shupe , Ben Rusholme , Frank Masci , Carrie Nugent , "Rebbapragada, Umaa D (398J)" , Adam Miller , "Mansi M. Kasliwal" , Eric Bellm , Tom Barlow , Lin Yan , physarah@gmail.com, Vandana Desai , Jason Surace , David Cook , Thomas A Prince , Yi Mei Cc: Steve Groom , Lisa Storrie-Lombardi Minutes (2 / 9 / 2016) Present: Dave S., Jason, Eric, Adam, Frank, Tom B., Tom P. * Lightcurve release preparation: analysis of corruption issue was on hold. To be revisited. Optimistic that problem can be identified, including “panic” error from relative photometry step. * Brief discussion on data structure/format of lightcurve(s) downloaded from UI (initiated by Jason; he also offered to draft some specs, thank you!). * Another discussion on whether to fix the lightcurve-duplication issue for the Fall release. Answer: not enough time to do so before then, but needs thought for ZTF. Problem is how to ensure consistent photometric corrections from the relative photometry step across fields/chips. * P(Z)TFIDE updates: Frank and Eran had discussion on whether we can add a switch to run the new "ZOGY” algorithm without too much disruption to the IDE flow. Frank says it’s possible. Discussions continue. * Halpha reprocessing: test calibration code runs, but some details need to be cleared up with Dave C. There are also plans to conduct some special Halpha experiments to assist with calibrations. Jason mentioned possibility of observing some strong narrow-lined Ha objects across z-range to try and map out RSR of Ha filter. * mention of iPTF experiment to support Gaia in the near future (Tom P. via telecom). Regards, Frank On Feb 9, 2016, at 8:47 AM, Frank Masci wrote: Hi All, Today 1PM, KS-300. Agenda: 1. Lightcurve release preparation and status. Short-term plan: - repair corrupted match files in ops (need resolution by Feb 15); - backup all updated match files to either ptfdepot, sbx5 or Cahill; - run relative photometry to compute/apply ZP corrections to match files in ops; - cross-check against backup copies (that didn’t have dZP applied); - remove backup copy; - freeze ZP-corrected (hdf5) match files intended for release by copying to separate partition; - revisit schema for IRSA DB loading and confirm load-file specs with Steve; - draft specs to support loading of “object”, “associations”, “dZP” tables for IRSA; - analyses and documentation: validation of ZP corrections (rms checks); - write software to pull apart hdf5 files and generate the various IRSA load-files. 2. Summary of recent discussions on lightcurve GUI and data-download capability/format. 3. P(Z)TFIDE developments. 4. ZTF Software & Deliverables document feedback. 5. Halpha reprocessing status. 6. fyi only: not this meeting, but concurrent: - bring online all 30+ ztfops nodes for prototyping; other DB/server hardware procurement - tentative plan: design/set-up operations-like DB; deploy photometric PL; streamline LC pipeline steps; redesign Transients DB and loading methodology; deploy realtime PL; ... Regards, Frank --- Goto Meeting: 1. Please join my meeting. https://global.gotomeeting.com/join/595527485 2. Use your microphone and speakers (VoIP) - a headset is recommended. Or, call in using your telephone. Dial +1 (213) 493-0622 Access Code: 595-527-485 Audio PIN: Shown after joining the meeting Meeting ID: 595-527-485 From: Frank Masci Subject: Re: iPTF/ZTF Meeting Today Date: February 2, 2016 at 3:25:56 PM PST To: Russ Laher , David Shupe , Ben Rusholme , Frank Masci , Carrie Nugent , "Rebbapragada, Umaa D (398J)" , Adam Miller , "Mansi M. Kasliwal" , Eric Bellm , Tom Barlow , Lin Yan , physarah@gmail.com, Vandana Desai , Jason Surace , David Cook , Thomas A Prince , Yi Mei Cc: Steve Groom , Lisa Storrie-Lombardi Minutes (2 / 2 / 2016) Present: Dave S., Lin, Jason, Eric, Russ, Ben, Adam, Frank, Tom P. * Lightcurve release preparation: corruption issue still being explored. A test case was identified to explore end-to-end (starting from "nightly file” to source-matching to relative photometry). Resolution/plan needed by mid Feb. * Related to future iPTF lightcurve public releases (excluding the one planned this summer/fall): decommissioning is likely to be extended until end of 2016 (Eric). Plan is to append new data to meta load files for final archival spanning all PTF/iPTF. Partners to provide funds for extended iPTF processing, archive tasks, and other requests. * PTFIDE updates: still in progress (Frank) * Halpha reprocessing: code ready for testing; ZPVMs need tweaking and may be ready next week (Dave C.). * Brief discussion on lightcurve UI capabilities: e.g., enable all-sky count statistics, period finding with periodogram output, phase-folded lightcurve output. All TBD. * Discussion on marshal (Cahill) dependency on ptf.caltech.edu proxy server. A DNS redirect(?) problem may have brought down communications (and all marshal activities) last weekend. Contact details are now available, although not sure where the problem occurred. Potential improvements for ZTF identified (Lin). Regards, Frank On Feb 2, 2016, at 9:42 AM, Frank Masci wrote: Hi All, Today 1PM, KS-300. Agenda: 1. Lightcurve release preparation and status. Short-term plan: - repair corrupted match files in ops (need resolution by Feb 15); - backup all updated match files to either ptfdepot, sbx5 or Cahill; - run relative photometry to compute/apply ZP corrections to match files in ops; - cross-check against backup copies (that didn’t have dZP applied); - remove backup copy; - freeze ZP-corrected (hdf5) match files intended for release by copying to separate partition; - revisit schema for IRSA DB loading and confirm load-file specs with Steve; - draft specs to support loading of “object”, “associations”, “dZP” tables for IRSA; - analyses and documentation: validation of ZP corrections (rms checks); - write software to pull apart hdf5 files and generate the various IRSA load-files. 2. ZTF Software & Deliverables document (in review). Frank to convene meeting. 3. P(Z)TFIDE updates (in progress) 4. Halpha reprocessing status. 5. fyi only: not this meeting, but concurrent: - bring online all 30+ ztfops nodes for prototyping; other DB/server hardware procurement - tentative plan: design/set-up operations-like DB; deploy photometric PL with hooks for abs.cal using PS1; streamline LC pipeline steps; redesign Transients DB and loading methodology (‘active’ vs ‘quiescent’ tables); deploy realtime PL; ... Regards, Frank --- Goto Meeting: 1. Please join my meeting. https://global.gotomeeting.com/join/595527485 2. Use your microphone and speakers (VoIP) - a headset is recommended. Or, call in using your telephone. Dial +1 (213) 493-0622 Access Code: 595-527-485 Audio PIN: Shown after joining the meeting Meeting ID: 595-527-485 From: Frank Masci Subject: Re: iPTF/ZTF Meeting Today Date: January 26, 2016 at 6:03:39 PM PST To: Russ Laher , David Shupe , Ben Rusholme , Frank Masci , Carrie Nugent , "Rebbapragada, Umaa D (398J)" , Adam Miller , "Mansi M. Kasliwal" , Eric Bellm , Tom Barlow , Lin Yan , physarah@gmail.com, Vandana Desai , Jason Surace , David Cook Cc: Steve Groom , Lisa Storrie-Lombardi Minutes (1/26/2016) Present: Russ, Dave S., Lin, Jason, Ben, Eric, Dave C., Adam, Frank * LC release preparations, short term plan: - match-file corruption issue identified (~0.5million pids affected), repairs in progress; - temporarily backup all updated match files to either ptfdepot or Cahill disk; - run relative photometry to compute/apply ZP corrections to match files in ops; - cross-check against backup copies; - remove backup copy; - freeze ZP-corrected (hdf5) match files intended for release by copying to separate partition; - revisit schema for IRSA DB loading and confirm load-file specs with Steve; - write s/w to pull apart hdf5 files and generate the various load-files. * Eric to look into getting help on validating LC files for release, performance of ZP corrections for documenting, etc.. * Russ analyzed impact of removing astrometry.net from camera splitting PL. Conclusion: need to keep for now until input TCS pointing is better understood, i.e., running this provides a better astrometric solution to seed SCAMP downstream, although there are details in SCAMP that are not understood either. Also, the telescope encoder-reported positions TELRA/TELDEC were replaced with the commanded position equivalents, OBJRA/OBJDEC to initialize the raw WCS. The latter show a much reduced scatter relative to TELRA/DEC. * Request to enable batch-like forced-photometry requests on PTF data (Lin). Infrastructure is mostly in place; input batch file format and s/w updates need to be scoped (Russ). * PTFIDE updates: still in progress (Frank) * Integration/reporting of streaks in MOPS and MPC reports: classifier code in progress at JPL. Expect delivery ~ end of Feb. * Discussions on latencies and throughput testing in links (kinks?) from summit to IPAC reinitiated with Patrick S., Roger S., and Andy B. Patrick: not yet ready to run stress test since still trying to understand activity at network level. * Halpha reprocessing: Dave C. presented example ZPVM maps for use in pipeline. Details on how to handle color-term dependencies remain for accurate absolute calibration. Experiments are planned in March to refine calibrations. * Adam M - working on new star/galaxy classification table built from reference image catalogs. Trying to understand biases in training set while validating classifier. * ZTF s/w requirements and deliverables document: Frank and Eric had breakout session. Eric to provide feedback (and augment) current bulletized writeup on wiki before Frank does a more formal writeup. Also discussed ancillary products to support public alert stream in year 3. Regards, Frank On Jan 26, 2016, at 9:37 AM, Frank Masci wrote: Hi All, Today 1PM, KS-300. Agenda: 1. Lightcurve release preparation and status (beta/experimental version, goal ~ 7/16) - repair of corrupted match files - draft specs to support loading of “object”, “associations”, “dZP” tables for IRSA - analyses and documentation: validation of ZP corrections? 2. Not this meeting but needs thought: redesign of LC pipeline (source matching and relative photometry) 3. ZTF Software Requirements and Deliverables document (in preparation) 4. P(Z)TFIDE updates (in progress) 5. Removal of astrometry.net from p7 (camera splitting pipeline): recap of findings 6. Moving-object pipeline: integration of streaks for MPC, status of ML classifier? 7. ZTF-like transfer and bandwidth testing from Palomar to IPAC: discussions underway 8. Not this meeting, but concurrent: - redesign of Transients DB and loading methodology: ‘active’ vs ‘quiescent/historic’ tables - Questions: realtime use cases/queries?; N-day rolling window for active tables?; negative subtractions? - bring online all 30+ ztfops nodes for prototyping; other DB/server hardware procurement plans 9. Halpha reprocessing Regards, Frank --- Goto Meeting: 1. Please join my meeting. https://global.gotomeeting.com/join/595527485 2. Use your microphone and speakers (VoIP) - a headset is recommended. Or, call in using your telephone. Dial +1 (213) 493-0622 Access Code: 595-527-485 Audio PIN: Shown after joining the meeting Meeting ID: 595-527-485 From: Frank Masci Subject: Re: iPTF/ZTF Meeting Today Date: January 19, 2016 at 3:58:21 PM PST To: Russ Laher , David Shupe , Ben Rusholme , Frank Masci , Carrie Nugent , "Rebbapragada, Umaa D (398J)" , Adam Miller , "Mansi M. Kasliwal" , Eric Bellm , Tom Barlow , Lin Yan , physarah@gmail.com, Vandana Desai , Jason Surace , David Cook Minutes (1/19/2016) Present: Eric, Dave S., Lin, Ben, Russ, Dave C., Adam, Tom, Jason, Frank, Umaa * Frank to call meeting with Dave S, Eric, Russ, others? to discuss match-file corruption issue and see what can be recovered; settle on products for release; alternative is to use Cahill ‘static’ products. If time permits, also discuss content/format of merge files for IRSA DB loading, accounting for future redesign (next bullet). * Call another meeting to talk about redesign of src-matching / light-curve pipeline. * goal is to have some test version of LC-retrieval interface in place for Summer School (~ 7/18/16); based on existing toolkits. Alternative is to have a static file of lightcurves/metadata for students to experiment with. * request to enable bulk downloads of LCs in future (Eric), method/format TBD. * PTFIDE upgrades are in progress (Frank). * tuning of astrometric calibration modules in progress. Dave C. to send Dave S. Halpha failures. * removal of astrometry.net from camera-splitting PL; impact not fully understood; in progress (Russ) * plan to bring on-line all remaining ztfops drones (31 + 1 spare?) by ~ end of Jan; bare metal only (Ben) * request to start experimenting with data-link(s) from Palomar to IPAC (Ben); bandwidth testing.. Frank already initiated discussions with Roger Smith and Richard Walters. * iPTF decomissioning: expect end of Dec 2016. * Halpha reprocessing: Dave C. to deliver new ZPVMs, calibration code(s), and processing plan with all i/o; recalibrate catalogs using SDSS overlaps. Stacking not possible at present within operations. Regards, Frank On Jan 19, 2016, at 9:17 AM, Frank Masci wrote: Hi All, Today 1PM, KS-300. Agenda: 1. Overview of path forward. 2. Lightcurve release preparation and status (beta interface ~ 4/15/16) - settle on content and format of match (merge) files to deliver to IRSA for loading - additional LC metrics. 3. Redesign of LC pipeline (source matching and relative photometry) 4. PTFIDE updates & subsequent RB retraining 5. Tuning of existing astrometric calibration; more robust verification metrics 6. Removal of astrometric calibration (astrometry.net) from p7 - camera splitting pipeline. 7. Moving-object pipeline: linking and integration of streaks, and reporting to MPC 8. Not this meeting, but concurrent: - redesign of Transients DB schema; batch loading; partitioning (active vs. quiescent tables) - throughput study on existing realtime pipeline and transients DB; external queries - bring online all 32 ztfops nodes for prototyping; other hardware procurement plans 9. Halpha reprocessing Regards, Frank --- Goto Meeting: 1. Please join my meeting. https://global.gotomeeting.com/join/595527485 2. Use your microphone and speakers (VoIP) - a headset is recommended. Or, call in using your telephone. Dial +1 (213) 493-0622 Access Code: 595-527-485 Audio PIN: Shown after joining the meeting Meeting ID: 595-527-485 From: Frank Masci Subject: Re: PTF Meeting Today (1/5/16) Date: January 5, 2016 at 4:55:36 PM PST To: Jason Surace , Russ Laher , Vandana Desai , Lin Yan , Tom Barlow , Eric Bellm , "Mansi M. Kasliwal" , Adam Miller , David Shupe , "Umaa D (398E) Rebbapragada" , Carrie Nugent , Ben Rusholme Present: Russ, Ben, Frank, Lin, Jason * Russ has been rerunning source-matching/relative photometry for 88 nights requested by Eric. There were failures for 53 out of 900 pointings (field/chip/filter combinations). These remain a mystery and the issue is currently being looked at, although see the next bullet. * There was a discussion on how to better streamline the light-curve pipeline. For example, have the source-matching step write to a separate file per pointing (other than hdf5), say in frameproc, then execute relative-photometry on that product. The latter could become the “load file" for IRSA. All is TBD since the existing software will be problematic for ZTF. * Related to the previous point, there was also a discussion on the hdf5 format. It's understood that the Galactic marshal currently uses these files (rsync’d by Eric) and therefore will continue to be generated in the same format. However, it wasn’t clear if this will still be the case for ZTF given there are plans to redesign the marshals. One thought was to query IRSA instead for light-curve products on some TBD timescale once the interfaces are worked out. * Upcoming light-curve data release: the following analyses/discussions need to happen: (i) which fields/pointings to exclude due to bad quality; (ii) light-curve query interface design and functionality. * Automated reference-image building infrastructure: this was a relic task that may no longer be needed. This was in the context of continuously regenerating references following acquisition of new data satisfying certain criteria. There is no longer a need (at least for iPTF) to generate deeper references. The plan is to continue to regenerate on a per-case basis where existing products are identified as low quality or absent. * Next hardware purchase: there are plans to purchase another 32 nodes in the next couple of months. This will give us at least 64 nodes to experiment with. Severs and additional disks can wait until we better understand the needs from ongoing performance analyses. Regards, Frank