vsa logo
VSA Home
Start Here
Data Overview
Known Issues
the Surveys
Coverage Maps
Schema browser
Data access
Login
Archive Listing
GetImage
MultiGetImage
Region
Freeform SQL
CrossID
SQL Cookbook
Q&A
Glossary
Release History
Gallery
Publications
Monitor
Downtime
Links
vsa logo bottom
IFA     ROE
Home | Overview | Browser | Access | Login | Cookbook 
  VSA logo

VSA - Known Issues

General problems:


Please pay particular attention to the following outstanding problems:

VVVXDR1 is missing Quality Bit flags for OB frames, i.e. vvvxDetection.ppErrBits=0 for all non-deep science frames, which means that deblends, saturation, bad pixels, regions in under exposed and from detector 16 are not flagged. We are currently fixing this and will update the release in stages, as we progress. Firstly we will update the Detection tables, which will mean that the light-curve statistics (vvvxVariability) and band-merged table (vvvxSource) will be out of sync. Then we will update the band-merged columns e.g. j_1ppErrBits and primary/secondary flags in vvvxSource and finally the variability tables. FIXED 20230224

From VMC releases VMCv20160311 onwards including VMCDR4 and VMCDR5, the ordering of the bands in the source table was reversed (YJKs --> KsJY), so that astrometric offsets would be computed from Ks, which had a deeper image. However, it has just been discovered that this lead to Y attributes in the Ks columns in vmcVariability and vmcVarFrameSetInfo since the code was not updated to use the same band pass class as the source merging. We understand which columns are affected now and will fix this as soon as possible starting with the latest team release and public releases and working backwards. FIXED 20201224

The vvvSourceXDetectionBestMatch is unavailable in VVVDR5 while it is being updated (2020-10-01). This is taking longer than expected, but should be completed soon. FIXED.

The MapApertureIDsultravistaMapLc table contains the list of apertures used for light-curves and should be selected using the specification in RequiredMatchedApertureProduct for programmeID=160 and mapID=2, but actually used the specification for programmeID=160 and mapID=1, the dual image mode selection. This leads to 451631 apertures selected from the Ks-band deep mosaic, rather than 576890 selected from a merged-band catalogue of YJHKs+NB118 deep mosaics. The code has been fixed, and this will be corrected in the next release.

The averaging in time intervals is usually done in pawprint pointings, so an object in a VISTA tile/mosaic will probably be on 2 or more pointings, so will have overlapping intervals. In some cases we have specified to sum over overlaps (see RequiredMapAverages, but usually in special cases with limited numbers of objects. This means that objects will not have the full signal to noise that they would have if averaged over all pointings. However, it averaging over the overlaps has the downside of keeping track of the inputs - the MapProvenance would not be suitable, so the table structure may have to change.

20190115. In some of the Value Added Catalogues (team generated catalogues), where primary/secondary sources were flagged across the tile overlaps e.g. vvvPsfDophotZYJHKsSource, vvvPsfDaophotJKsSource and vmcPsfSource, the priOrSec flagging was incorrect and flagged many objects within the main tile away from the boundaries. The code has been fixed and the tables will be updated shortly. The VMC table has already been fixed. All tables have been fixed (2019-04-15)

The vvvPsfDophotZYJHKsSource is unavailable in VVVDR4 while it is being updated (2019-03-19).

20181101. Galactic coordinates (l,b) calculated for value-added catalogues (e.g. vvvDophotZYJHKsSource) are incorrect, because they used a Python function that assumed an input in B1950 coordinates. This may also affect some recent external catalogues which do not calculate l,b themselves. The main pipeline tables \ (vhsSource, vvvSource etc) are not affected because their l,b coordinates were calculated from fast C/C++ code. Over the course of the next few days, we will list all the affected tables, and within a few weeks we expect to fix this problem. All other coordinates: ra,dec, Cartesian are unaffected. Neighbour tables, that are created using ra,dec are equally unaffected. All fixed, (2019-04-15) 20190115. All of the code has been fixed and tested and fixer code is in place to update existing tables. The tables that need updating are vvvDophotZYJHKsSource (also has other issues), vmcPsfDetections/Source (fixed), vvvProperMotionCatalogue, vvvParallaxCatalogue. All fixed (2019-04-15)

Problem with grouting of OB tiles between Feb 2015 and April 2016
CASU have identified an issue with photometry of tiles produced between Feb 2015 and April 2016 inclusive. At the grouting stage the pawprint zeropoints were not applied correctly to the calculations, so the fluxes in the tiles are incorrect. CASU have reprocessed the data, which will be called v1.3.1. WFAU are in the process of ingesting the reprocessed data, and releases from November 2016 onwards will include the replacement tiles and catalogues. Fixed

Incorrect band-merging in vvvSynopticSource
When developing code to merge vvvSource and vvvSynopticSource into one table for future releases, it was found that 4 pointings in vvvSynopticSource had the incorrect Ks band frame assigned. The logic for merging looked for complete OBs (JHKs), but in some cases a J or H frame was missing. Another problem was due to searching from the pointing centre in too small a radius. It was also noted that one pointing has only Ks-band frames. This occured because later Ks band frames had a 6' offset to earlier ones and were grouped incorrectly.

We have corrected the Ks-band assignment logic and recreated the vvvSynopticMergeLog and vvvSynopticSource tables and also the vvvSourceXSynopticSource linking table. These will shortly be added back into the release. Fixed 20151217.

Effects of Astrometric Library update May 2014 - Oct 2014
Noting that no VVV tiles seem to have been flagged for bit=12 during our current processing, we tracked the problem down to the updates to the 3rd party AST library from Starlink.

New Features in AST

Support for Distorted Projections: * SCAMP -TAN: This is the same as the TPV projection, but uses a CTYPE code of -TAN instead of -TPV. AST differentiates between SCAMP (Bertin 2006) TAN headers and standard TAN headers by looking for PV keywords attached to the latitude axis (a standard TAN projection should have no such latitude PV keywords).

When we run these codes from data in the data base we had supplied all the WCS keywords whether they existed or not (with default values if not), and in the past the PV keywords were ignored from images with the TAN projection. Unfortunately, to deal with the two types of TAN projection, this is no longer the case. While it seems dangerous to use the same CTYPE for two different projections, we should have been more careful about what we supplied to the AST code.

We have looked through the code and worked out the places that will suffer from this problem. The ppErrBits flag will be incorrect or missing for bit=12 and bit=24, which match the position of a pawprint onto the tile. The ppErrBits has a knock on effect on the primary/secondary selection in the Source tables, so there will be sub-optimal determination of primary sources for the parts of tile that overlap with detector 16 from one or more of the pawprints or where a pawprint detector has been deprecated. Similarly the data in theAstrometricInfo tables will be unreliable in places. This is necessary for completing the SourceXDetectionBestMatch tables, which links unique sources in the Source table to epochs. Specifically epochs where there are no-detections may be incorrectly set or flagged, so if you are interested in the parts of light-curves where there isn't a detection be careful.

The following released databases are affected:

  • VIKINGv20140530 --> the ppErrBits for J-band deep tiles and vikingSourceXDetectionBestMatch / vikingVariablity. The bit 12 flags are missing for the J-band deep tiles. This release will shortly be superseded by a new P92 release
  • VMCv20140903 --> All the P92 tile data, the deep tiles and vmcSourceXSynopticSourceBestMatch / vmcVariability. The bit 12 flags are missing or in-correctly applied for the deep tiles and P92 OB tiles. This release will be re-created to both resolve this issue and to deliver a new optimised pairing radius for source merging as requested by the PI.

Poor photometry in some VMC deep tiles
One Y-band tile (1_1, multiframeID=2768124) had a zeropoint that was ~0.5 mag lower than expected. Further investigation showed that the magnitudes were approximately correct because the aperture correction was also ~0.5 mag larger than expected. The seeing seems to have increased from 1.3" for the pawprints to ~1.7" for the tile. However, when the Y-J colours were plotted as a function of position for this pointing, one pawprint stood out as having colours that were 0.2 mag offset from the other parts of the tile.

Looking at the inputs, we found they were exactly the same as those for the 1_1 Y tile (multiframeID=1403990) in the VMCv20130304 release. Just in case, we did a couple of experiments removing some of the later inputs, particularly one odd pawprint (20110916_v1.2/v20110916_00513_st.fit) that is the only component of an incomplete tile. However, this did not make any difference.

Further experiments demonstrated that older versions of the CASUTOOLS software (e.g. casutools-1.0.21) produced tiles like multiframeID=1403990, and newer versions gave tiles like multiframeID=2768124. Experiments narrowed the difference down to the nebuliser or mosaic package, and a change in casutools-1.0.22 or casutools-1.0.23.

It should be pointed out that the data that went into this tile did have quite a range of seeing, from 0.85" to 1.53", so in the future, eliminating data with seeing>1.1" or seeing>1.2" may reduce this issue. While changes in the mosaicing code may have made this particular tile with highly variable seeing worse, the same changes have improved the quality of many other tiles.

Some more experimenting needs to take place to definitely say that the variable seeing is the problem and to determine what ranges give good tiles.

Zeropoint miscalcuation issue
An anomaly was discovered in the VMC photometry between the VMCv20110909 and VMCv20121128 releases. This was caused by a difference with the zeropoints: the fluxes or other corrections are essentially the same. By comparing the deep photometry to the OBs we determined that the error occured in the VMCv20121128 release.

The problem occurred in the ZP calculation of deep stacks - we compare the new deep stack/tile catalogue photometry to the components - OB stacks in the case of deep stacks, or deep stacks in the case of deep tiles. Originally we took the OB stack photometry from the archive, but found in the case of the VVV this took all the archive resources, so we switched to taking the OB stack photometry from the FITS catalogues, as we already did for the deeps. However some of the corrections, which are set to 0 for deeps were not applied for these OB frames. This was realised shortly afterwards and corrected, but we forgot to check all the existing releases, including VMCv20121128. The ZP offset is typically 0.02 mag. The current code uses the archive to get the OB stack photometry except in the case of the VVV. Differences between using the FITS catalogues and the archive are now <0.001 mag, and these difference come from stricter cuts in the archive selection, which makes use of the ppErrBits.

We also discovered that some of the Ks-band OB frames had been recalibrated at the time of processing data for VMCv20120126, by a similar amount. I think that the ZP of the deep tile was miscalculated by the process above and then the recalibration procedure, which was designed to make small adjustments to improve light-curves fed the ZP corrections back into the OB level stacks and tiles. We turned off the OB recalibration stage over a year ago when we found it was making the VVV photometry worse and not making a significant improvement to the VMC photometry. At the time, I thought this was to do with the flatness of the tiles, but now I think it might have been because of some mistakes we made in the past. I will do some tests with the 1.3 data and then turn it back on again in future releases, if I am convinced that there are no problems and after I have introduced a couple of tests which will pick up the above problems early.

Unfortunately VMCv20130304 will be partially affected by these problems. We used existing deep stacks and tiles where possible and any pointings where there were no new data will be affected. Any with at least one new epoch should be correct, as the ZP will have been recalculated using all the corrections.

VHS does not have any deep components, so none of these issues affect it. VIDEO produce there own deep mosaics, so these issues do not affect it either. Some of these effects had already been discovered in the VVV originally, leading to the code changes - the OB zeropoint issue was reverted in VVVDR1 and the deep stacks and tiles for the next release are unaffected. VIKINGv20121205 and VIKINGv20130417 are also affected. The problem has been fixed in the VMC, with a new version VMCv20130805

Corrupted files on disk38
Following a hardware isssue a small fraction (10 percent) of FITS files on disk38 are showing as corrupt. We are working to identify and fix the files but in the meantime please be aware of the issue if you are downloading files from disk38 (the disk name shows in the URL of the download file). Affected image files will have some extension(s) that will fail to uncompress or open. We are not sure of the effect on catalogue files.

Poor photometry in some VVVDR1 Ks deep tiles (vvvSource Ks band)
As a result of not ordering frames of differing exposure times prior to stacking, the resultant tileDeepStacks suffer from incorrect photometry in the various pawprint offsets. Magnitudes can be off by around 0.6 mags in the worse cases. Note the Ks photometry in the vvvSource table often comes from the tileDeepStacks. The disk fields are most affected as typically the initial observations had different exposure time to the repeats. Bulge fields have been observed using the same exposure times and are OK. Of the 336 framesets in vvvMergeLog with Ks frames, 114 are likely to show the problem to varying degrees, list of affected frames
November 2012 fixed - columns in vvvSource and vvvDetection that originate from the affected deep tiles have been reset to defaults.

Deep tiles not available
Currently tiles are available from single observing blocks, but are not available across multiple OBs. The software to produce these is still undergoing some development and testing, but should be available shortly. This will affect surveys which require deep tiles: VVV, VMC, VIKING. VHS is mainly composed of single epoch (OB) data, and VIDEO and ULTRAVISTA will produce their own mosaics. In the meantime VVV, VMC and VIKING will be curated using pawprint data. This means that all the band-merged tables, synoptic and variability tables will be produced using pawprint data. However the single epoch tile data will be available in the release database too, it just won't be matched up like the pawprint data. February 2011 fixed - all subsequent releases have deep tiles

Accurate mjds not available
Tiles in CASU version-1.0 data did not have mjds calculated using the grouting software and therefore the only mjds available were through the mjd-obs (the starting mjd of the observation). Version 1.1 data will have mean-mjds calculated for all tiles. We will also calculate a mean-mjd for pawprints and add a new attribute mjd to the Detection tables. July 2011 fixed - all subsequent releases have correct detection mjds

Difficult to use tile-pawprint table
The first version of the TilePawPrint table that links tile detections to pawprint detections is difficult to use in conjunction with the Source table or SynopticSource. The problem is with the default entries added, with the selection of primary key, but mainly due to the fact that entries are allowed when there is no tile detection, but one or more pawprint detections. Since these may be of use for some users too, we have decided to keep them, but create a view (TilePawTDOnly) without them. Aug 16th 2011 fixed - all subsequent releases have these changes

Default mjd values in the SynopticSource table
It was found that the first SynopticSource tables produced with version-1.1 data had a bug which just added default values to the zmjd, ymjd, ... ksmjd columns. Sep 9th 2011 fixed - all subsequent releases will have these changes

Missing detections in VMC Ks Deep Tiles
It has recently been found that there are missing Ks band detections in some VMC deep tiles, in a grid like distribution. This grid distribution matches the areas of overlap of all six pawprints in the tile, where the confidence image should have maximal values. However, it was found that the confidence in these regions was negative, leading to a non-detection. Further tests revealed the source of the problem: the science images had been scaled to store the floating point data as integers so that the images could be losslessly compressed and this scaling was inadvertently applied to the confidence images. The BSCALE values depend on the NDIT and NJITTER. The NDIT values are particularly high for the VMC Ks band images, and the BSCALE is correspondingly lower, so it is only the VMC deep Ks tiles that are affected. This affects tiles with smaller values of BSCALE (MultiframeDetector.datascale) in all VMC releases upto VMCv20120127. This will be fixes in subsequent releases.

Flagging of saturation in tiles.
In the VVV, a large number of additional variables has been found at the bright end. Further analysis has shown that using a single saturation cutoff level for a tile does not work well, different parts of the tile come from different pawprints with different saturation levels and this needs to be factored in.

Incorrectly deprecated Ks band VVV tiles in VVV-DR1.
191 of the Ks band OB tiles have been deprecated. These were deprecated when we attempted to recalibrate the tiles by comparing the photometry to the deep Ks band tiles. Our code automatically adjusted the zeropoint unless the difference was >0.05 magnitudes, in which case it was expected that the frame had a significant problem and the frame was deprecated (see Cross et al. 2009, MNRAS, 399, 1730 for a discussion and examples of the code). When we ran the code on the VVV we realised that the process was not working well for the VVV - too many frames were being deprecated and the RMSs of the non-variable stars did not improve, in fact they became worse. We reversed the process, changing the ZPs and magnitudes back to the old values and undeprecating the frames. However, it seems that we reversed the deprecations in MultiframeDetector, but not in Multiframe.

These 191 tiles were not included in the vvvSourceXDetection and vvvSourceXDetectionBestMatch tables, the latter is used for creating light-curves, nor were they included when calculating the variability statistics in vvvVariability.

This problem should not occur in the future. The recalibration code has been turned off for tiles in the VSA, and the deprecated frames have now been undeprecated. In agreement with the PIs, this problem will not be fixed in the VVVDR1 release (since this will take a couple of months to fix), but will be fixed in subsequent releases.



Home | Overview | Browser | Access | Login | Cookbook
Listing | FreeSQL
Links | Credits

WFAU, Institute for Astronomy,
Royal Observatory, Blackford Hill
Edinburgh, EH9 3HJ, UK

vsa-support@roe.ac.uk
28/2/2023