WFCAM SCIENCE ARCHIVE

UHSDR1 initially contained incorrect values for mergedClassStat,pStar, pGalaxy, pNoise. They were out by a factor of root 2 as the number of bands had been set at 2 rather than 1. These attributes were fixed 2019-11-06.

Poor astrometry for some GPS frames in DR8
A small number of J and K GPS frames have poor astrometry in DR8

The issue lies with the WCS header of the catalogue that was used to calculate the RA and Decs of the objects in gpsDetection and gpsSource. The GPS re-processing for DR8 saw the updating of the PV2_3 value depending on the waveband. Previously all wavebands had PV2_3 set to -50. The revised values being: J band, PV2_3 = -46 and K band, PV2_3 = -60. Although the headers of the affected files contain the revised values the rest of the WCS was not updated (i.e. corresponds to PV2_3 = -50). Consequently positions in the affected frames are calculated incorrectly and get worse the further the position is away from the centre of the pawprint. At the far corners of a pawprint the positions are out by 0.4-0.5 arcsec in J and 1.0-1.1 arcsec in K
Further details - including list of affected frames

Corrupted files on disk38
Following a hardware isssue a small fraction (10 percent) of FITS files on disk38 are showing as corrupt. We are working to identify and fix the files but in the meantime please be aware of the issue if you are downloading files from disk38 (the disk name shows in the URL of the download file). Affected image files will have some extension(s) that will fail to uncompress or open. We are not sure of the effect on catalogue files.

Issues affecting earlier releases but fixed in more recent release(s):

UKIDSS DR7+ LAS and GCS and DR6+ GPS have a small number of frames (around 0.1%, or a few multiframes in each passband) missing in the source merging process; corresponding rows are missing from the merge log and source tables. For a given passband and survey, the following SQL will show the detector frames missed - in this case, 4 Y detector frames missed in the LAS:
SELECT d.multiframeID, d.extnum
FROM Multiframe m, MultiframeDetector d LEFT OUTER JOIN lasMergeLog l
on l.ymfID=d.multiframeID AND l.yenum=d.extnum
WHERE m.filterID=2 AND project LIKE '%las%' AND frametype='stack'
AND m.multiframeID=d.multiframeID AND m.deprecated=0
AND d.deprecated=0 AND (l.yenum IS NULL)
The problem is fixed for LAS and GCS in DR8 and all subsequent releases; it did not occur in DR6 and before; the problem is fixed for the GPS in DR7 and all subsequent releases; it did not occur in DR5 and before.
UKIDSS DR5+ contained an inconsistency between the dxsDetection photometry and the zeropoints in MultiframeDetector for intermediate (not deep) stacks. This was caused by an updating error on MultiframeDetector. This has been fixed and updated at the same time as the release of the UDS-DR5 data so this is no longer a problem for UKIDSS DR5.
UKIDSS DR4+ contained a small amount of deprecated data in the *Source tables due to (we believe) reprocessed data arriving at the archive end after source merging for a data release had been done. For example, the LAS contained 12 H-band and 6 J-band detector frames which were reprocessed and automatically deprecated so their detections did not appear in lasDetection; nonetheless the merged sources containing those detections slipped through into the release. We thank Richard McMahon for bringing this to our attention. The problem has been fixed in DR5+ and all subsequent releases; moreover it did not affect DR3+ and prior releases.
206 fields (204 GPS, 1 LAS and 1 GCS) have astrometric errors of order 1 arcsec. This problem is mainly confined to some of the most crowded star fields at Galactic longitudes l < 30 degrees, and is due to a minor procedural problem at the pipeline processing stage in the data flow system. A list of multiframe files affected is available here (kindly provided by Mike Irwin at CASU). As a consequence, sources that were detected in multiple passbands in those fields may not have been successfully paired up in the gpsSource catalogue, since the merging process uses a source matching radius of 1 arcsec in the GPS. The affected fields will be fixed in DR4, but they remain in DR3 and earlier releases. Thanks are due to Phil Lucas and Reba Bandyopadhyay for identifying this problem.

Error in dither offset calculation - In the DR3, DR2 releases the dither offset calculation that defines the underexposed region of stacked images was incorrect in some LAS and DXS passbands. This resulted in fewer sources, in these surveys, being flagged with bit 22 and to a lesser extent also bit 19 in the ppErrBits column.

Error in Gaussian sigma calculation - In all releases prior to DR4 there was an error in the Gaussian sigma calculation in the udsDetection catalogues from deep stacks (i.e from SExtractor output). In previous releases, we calculated sigma=sqrt(A_IMAGE**2+B_IMAGE**2). While testing a seeing and average stellar ellipticity calculator, this was found to be incorrect. The correct equation, derived from the equations in the SExtractor manual, is sigma=0.5*sqrt(A_IMAGE**2+B_IMAGE**2). This has been updated in the curation software and will be corrected in future releases. Anyone using the gaussian sigma in the UDS deep stack catalogues should correct this. In future releases, the UDS deep stacks will have seeing and average stellar ellipticity calculated too.

Incorrect values for variableClass in DXS Data Release 6
8th January 2010 fixed - DR6 was updated and the problem fixed
Adding new attributes to the Variability table led to an indexing error, which caused all the variableClass values to be incorrectly calculated, so that all have values of 0. The jvarClass and kvarClass values are correct though, so these can be used to discriminate between variables and non-variables. This problem will be fixed and the correct values released when the GPS and UDS are appended to Data Release 6.

Photometry issue with processed WFCAM data, observation dates from 12/03/08 (08A/B)
23rd March 2009 fixed - The affected files (stack images and catalogues) have been deprecated and replaced by version 2 (V2).
An error in flat-file catalogue photometry has been discovered that affects all version 1 processed data having observation dates from 12th March 2008 (semester 08A) onwards. The error has been traced to a bug in the standard nightly pipeline source extraction software.
Any users, of both survey and non-survey (i.e. private PI-led programmes), who have downloaded data observed from 12th March 2008 should be aware that there are systematic errors of 10% or more in various flux measures (e.g. aperture fluxes) in all catalogues in all wavebands. Image file pixel values are not affected by the problem, but there will be small (1% to 2%) systematic errors in the catalogue-derived zeropoints copied into the image headers. Please note that UKIDSS releases up to and including DR4, and PI programmes up to and including only data to 11/03/08 in Semester 08A, are NOT affected by the problem in any way.
VDFS operations at CASU and WFAU apologise for the error and the delay in delivery of UKIDSS Data Release 5, which will now slip by around 6 weeks. Over the first 3 weeks, the flat file products will be reprocessed, transfered and ingested as version 2 to replace the affected (version 1) products. 13th October 2009 (DR6) fixed for DXS, LAS, GCS - we have applied a correction written by CASU to redistribute the fluxes of the components according to their isophotal fluxes.

Erroneous extended source fluxes/radii/mags for deblends
A pipeline processing bug has resulted in all CASU processed catalogue deblend components having erroneous values of Petrosian, Kron and Hall fluxes, radii and magnitudes. The problem does not affect those attributes for isolated (un-deblended) catalogue detections. Deblended objects can be identified via the ppErrBits attribute: the SQL construct WHERE ... AND (ppErrBits&0x10) > 0 can be used to filter such sources. Users interested in extended source fluxes for deblended components should use the isophotal values for the time being; at the time of writing (November 2008) we are planning to fix this problem in DR6 et seq.

Badly stacked DXS intermediate stack.
A problem has been noted with a DXS intermediate stack in the ELIAS-N1 region. This stack was incorrectly stacked in extensions 4 and 5, for reasons unknown and has been missed by quality control so far. In extension 4, there are 3 images for each object (of similar brightness), and in extension 4 there are two images (one bright and one faint). Extensions 2 and 3 appear to be fine. This image has multiframeID=232788. The intermediate stack is one of several that are components of deep stack with productID=95, pointing EN011_1. This deep stack has been released in all data releases apart from DR1, where tougher than optimal seeing constraints rejected it. These deep stack images look fine, and any additional objects have probably been clipped out. Since the zeropoints are calculated on an extension by extension basis, most objects in these frames will have the correct magnitude, as the incorrect weighting will be compensated for by a slight change in the zeropoint from the expected value. For instance in the DR4 deep stack there are 17 input intermediate stacks. Extension 4 of multiframeID=232788 has 3 objects for each real source, and the brightest of these is ~1 magnitude fainter than expected. Thus the total flux will be 16.4 times a single input rather than 17 times, so each object would be ~0.04 mag fainter than expected. Instead the zeropoint will be ~0.04 mag brighter than expected. However, some objects at the edges of the deep stack which do not lie on multiframeID=232788 because of the dither pattern will be ~0.04 mag brighter. The deep stacks which are affected are: UKIDSS-EDR: multiframeID=249019, filename="20060123_v1/e20060123_000953_dp_sf_st.fit". This deep stack contains 9 intermediate stacks. UKIDSS-DR2: multiframeID=1104105, filename="20070131_v2/e20070131_0010400095_dp_sf_st.fit". This deep stack contains 17 intermediate stacks. UKIDSS-DR3: multiframeID=1624039, filename="20071108_v3/e20071108_0010400095_dp_sf_st.fit". This deep stack contains 17 intermediate stacks. UKIDSS-DR4: multiframeID=2012258, filename="20080612_v4/e20080612_0010400095_dp_sf_st.fit". This deep stack contains 17 intermediate stacks. The intermediate stack will be deprecated in UKIDSS-DR5 and a new deep stack will be created excluding this frame.

Unresolved problems:

Illumination correction bug, 08A - 10A
An error in the application of the illumination corrections has been spotted, affecting data from 08A until 10A inclusive. Also reprocessed data that has been ingested between the beginning of 2009 and 2011/03/15 is affected. All the data has been corrected with the H band illumination correction instead of the illumination correction of the respective band which might lead to an error of ~0.04 mag in the worst case but should be much smaller in general. Please note that the resulting systematic errors only really affect a small percentage of photometric measurements for brighter objects where the random errors do not dominate the effect.
This error will be fixed in the final data release.

GPS Photometric Calibration - Fixed in DR8
Improvements to the GPS photometric calibration have been made and the calibration procedures are now reliable in all parts of the survey. Analysis of the overlap regions between independently calibrated adjacent arrays shows that ~90% of adjacent fields have calibrations consistent to within 0.03 mag. We find that <0.1% of adjacent arrays differ by >0.1 mag. This is turn implies a high degree of uniformity in the calibration of 2MASS in the Galactic plane (since UKIDSS calibration derives from 2MASS) which has not previously been documented. Any remaining very rare instances of bad calibration are more likely to be due to bad data that slipped through quality control rather than problems in the calibration procedures.

A calibration problem in a small fraction of GPS fields was previously noted on this page, affecting DR7 and all previous releases. Sources in fields with anomalous zero points were flagged by setting a ppErrbits parameter to 131072 (or higher) in the gpsSource table in DR7. Further investigation showed that there were two calibration problems. (1) The photometric transformation from the 2MASS system to the UKIDSS system contains a term to describe the effect of interstellar extinction on the transformation of the 2MASS calibrator stars (see Hodgkin et al.(2009, MNRAS 394, 675). This empirically derived term was not accurate for J band observations in fields with high extinction throughout, owing to the limited amount of data available in such fields at the time the paper was written. An improved extinction term has now been established, following the procedures described in that paper. The selection of 2MASS calibrator stars has also been modified slightly to ensure that relatively blue dwarf stars are used whenever possible. These changes improve the accuracy of the zero points and source magnitudes in all three passbands. (2) The second problem was a modest (approx. 0.05 mag) underestimation of the aperture corrections in very crowded fields, which affected most fields in the GPS southern strip at longitudes l<15 degrees). The aperture corrections are an additive term in the computation of the zero points, so this led to an underestimation of the zero points at the same level. Fortunately, the aperture corrections are subtracted when computing the aperture magnitudes of sources, causing the error to cancel out almost perfectly. The net effect of this problem was therefore negligible for most users. Only users who were using the erroneous zero points in their own photometry would have been affected.

To put these changes into perspective, the typical change in source magnitudes between DR7 and DR8 was <0.01 mag. We find that the calibration of 99.8% of GPS fields has changed by <0.1 mag, and 98.6% of GPS fields have changed by <0.05 mag. In the southern fields at longitudes l<15 degrees the problems were a little more serious, leading to typical absolute changes of 0.016, 0.015 and 0.006 mag in the J, H and K calibrations respectively. In this southern portion of the survey the calibration of 1.3% of fields has changed by >0.1 mag, these significant changes occurring only in the J and H bands.

Default values for the photometric statistics in non-surveys
A software upgrade to numpy led to an error in the calculation of the best aperture for photometric statistics in the Variability tables. Therefore all the photometric statistics in Variability and the noise properties in VarFrameSetInfo are all set to default values. This affects all datasets released from 25th April 2009 until 7th May 2009.

Incomplete Best-Match tables for multi-epoch data sets
An error in matching the Source table to Detecion tables to create the SourceXDetectionBestMatch (BM) table has been discovered (5th May 2009). For sources where there are less than 3 good matches (3 different epochs), there are no entries in the BM table (except occasionally in dense regions). This was implemented to avoid calculating variability statistics for sources with too few detections. However, the side affect of this is to miss important links between frames, particularly for faint objects near the detection limit of individual epoch images, or datasets with a small number of epochs. We have modified the code and the BM table will include links between the Source and Detection tables for sources with fewer than 3 epochs but only calculate the astrometric statistics with 3 or more observations. We will apply this to the DXS in DR6, and the UDS in the DR5. PI programmes with multi-epoch observations will have this fix if they were released after 5th May 2009, and won't if they were released before. They will be fixed in future re-releases of data sets.

Under estimate of Kron and Petrosian fluxes for large galaxies.
The apertures that for Kron and Petrosian magnitudes have an upper limit of 12 arcsec to avoid incorrect background subtraction for galaxies where the size is greater than the background estimation. However this will lead to an underestimate of the flux of large galaxies. The very brightest galaxies, where background subtraction would not lead to significant errors, do have larger apertures.

Error in Kron radii
A bug has been identified in the Kron radii produced by the standard pipeline extractor. Therefore all catalogues except the UDS catalogues are affected. The exact nature of the bug has not yet been ascertained, only that the Kron radii are not being reliably measured. This will affect the Kron magnitudes, which should not be used until this bug has been fixed. The same tests have shown that the Petrosian radii are reliable and so for the foreseeable future, Petrosian magnitudes should be used for total magnitude estimates in extragalactic work.

Cross-talk images
The cross-talk artefacts produced by bright stars result in spurious souces in the object catalogues and tables. see CASU for description and examples.
Releases prior to DR1 all suffer from this problem. Semester 05B data in DR1 (but not 05A data) are less affected as improvements in the pipeline processing have reduced the effect. Starting with DR2, during post-processing quality error bit flags were set that should help with some of the issues e.g. cross-talk (see quality error bits page).

Persistence images
Can result in spurious souces in the object catalogues and tables. see CASU for description and examples; a more recent report is available here.

Concerning image processing, the CASU pipeline processing pages contain much information on known issues and image artifacts in WFCAM data - see, for example, here. For general help, support and any bug reports concerning CASU pipeline processing, please email casuhelp@ast.cam.ac.uk.

General archive modification history:

February 21st 2008: problem with FIRST cross-matches
We discovered a problem with the neighbour table, lasSourceXfirstSource whereby the pairing had been done incorrectly. The cross-matching was re-run and the table has been corrected in all UKIDSS release databases. Prior to this any cross matches queries executed in the EDR, DR1, DR2 or DR3 with FIRST making use of this table would have been in error.
January 7th - 12th 2008: missing pixel data
One of the disks holding pixel data was lost, consequently there will be no access to images that the disk held. See the list of affected dates. The data are being re-transferred from CASU. Note this only affects pixel access (eg getImage, multiGetImage) and downloads of flat files from the affected nights. Database queries are unaffected.
October 31st 2005: USNO-B database corrupted
The USNO-B database held in the archive was found to be corrupted and will remain unavailable until further notice.

WFAU, Institute for Astronomy,
Royal Observatory, Blackford Hill
Edinburgh, EH9 3HJ, UK

wsa-support@roe.ac.uk
13/11/2019