CCTV and imaging technology
Evaluations
HOSDB run regular i-LIDS performance evaluations. Manufacturers meeting the highest level of performance classification are entitled to use the trademarked i-LIDS logo in their trade literature.
Video Analytics (VA) systems may be submitted for evaluation in any of the five scenarios, and for either an 'Operational Alert' or 'Event Recording' role. In the Operational Alert role systems provide real-time detection of suspicious events to be dealt with by a human operator. In the latter systems act as a trigger for recording suspicious events and the recordings are analysed post event.
For the event detection scenarios the VA systems performance using a combination of the detection and false alarm rate is calculated using a measure known as the F1 score. A recall bias weights this metric to determin the relative influence of a systems detection rate versus its false alarm rate. This bias is set by HOSDB to represent the UK government and Critical National Infrastructure (CNI) requirements.
Systems for the Event Recording role are assessed using a higher value of recall bias as the detection rate is more important. In the Operational Alert role false alarms are a more significant problem and so a lower value of recall bias is set to a lower value.
Systems which have been evaluated will be awarded one of five possible classifications.
Evaluation application dates
The evaluation schedule until 2011 is:
-
Doorway surveillance - February 2010
-
Sterile zone - April 2010
-
Multiple camera tracking - July 2010
-
Abandoned baggage - October 2010
-
Parked vehicle - November 2010
-
Doorway surveillance - February 2011
Reminder notices will be issued to owners of i-LIDS datasets, indicating application submission deadlines for forthcoming evaluations. The application submission deadlines for the 2009-10 and 2010-11 financial years are as follows:
-
Sterile zone - 17 March 2010
-
Multiple camera tracking 9 June 2010
-
Abandoned baggage - 1 September 2010
-
Parked vehicle - 20 October 2010
-
Doorway surveillance 12 January 2011
Manufacturers of systems that have been selected for evaluation will be asked to sign an evaluation agreement containing the terms and conditions for the evaluation. For a copy of the agreement email the i-LIDS team.
Submission process
Manufacturers wishing to submit a VBDS for a forthcoming scenario trial should complete the evaluation application with:
- the details of their system
- its performance on the test dataset in the scenario; and
- role for which they wish to be evaluated.
The form should then be mailed or faxed to HOSDB.
After the deadline for application submissions, HOSDB will ask the manufacturers of selected systems to agree to a full set of terms governing the trial. These terms require the manufacturer to provide their system on free loan for the duration of the trial. Systems should be provided optimised ready for the scenario and role in question, with one channel set up to monitor each camera view (stage.) Provision must be made for HOSDB to irretrievably delete any video recorded by the systems during the trials.
Systems demonstrating a sufficient overall F1 value across HOSDB's private evaluation dataset, with no significant weakness in any particular subgroup of imagery (certain stages, weather conditions etc.) will be recommended for Government and CNI use in that scenario and role, subject to the terms of the trial.
After the trials, HOSDB will provide the manufacturers with feedback on the performance of their system(s), any weaknesses found and whether they are to be recommended for Government use.
Contacts:
The i-LIDS team
Home Office Scientific Development Branch
Langhurst House
Langhurstwood Road
Horsham
West Sussex. RH12 4WX.
Email: i-LIDS@homeoffice.gsi.gov.uk
Voicemail: (+44) (0)1403 213823
Fax: (+44) (0)1403 213827 - please mark ‘FAO: i-LIDS team’
