SITIS Archives - Topic Details
Program:  SBIR
Topic Num:  N07-018 (Navy)
Title:  Variable Remapping of Airborne Imagery
Research & Technical Areas:  Air Platform, Information Systems, Sensors

Acquisition Program:  PEO(T), PMA-265
 RESTRICTION ON PERFORMANCE BY FOREIGN CITIZENS (i.e., those holding non-U.S. Passports): This topic is “ITAR Restricted”. The information and materials provided pursuant to or resulting from this topic are restricted under the International Traffic in Arms Regulations (ITAR), 22 CFR Parts 120 - 130, which control the export of defense-related material and services, including the export of sensitive technical data. Foreign Citizens may perform work under an award resulting from this topic only if they hold the “Permanent Resident Card”, or are designated as “Protected Individuals” as defined by 8 U.S.C. 1324b(a)(3). If a proposal for this topic contains participation by a foreign citizen who is not in one of the above two categories, the proposal will be rejected.
  Objective:  Develop, in real time, a predicted target image based on imagery from one or more different sensors, taken at different times and from different perspectives.
  Description:  Unmanned air vehicles (UAVs) are rapidly emerging as an important adjunct source of targeting information. They can be employed close to the target in harm's way to provide targeting quality imagery to other strike platforms carrying weapons. However, there needs to be a positive handoff to the strike platform. In this case, the positive identification (ID) is made by positively associating the ID quality imagery obtained by the unmanned aircraft system (UAS) with the imagery being obtained by the strike platform’s imaging sensor. Often, geopositional confirmation is not possible or cannot disambiguate the target vehicle from others in the immediate vicinity. In these cases a visual correlation must be made. To effect this correlation with high confidence, it is important to present the targeting imagery as if it were being viewed from the weapon platform, which may be observing the target from a significantly different perspective. In recent years there has been an extraordinary amount of research in computer vision to render images from a moving camera, or an array of cameras, to create a predicted or synthetic image from another perspective with high fidelity. The technical innovation required is how to use a combination of image warping techniques, sensor modeling, atmospheric modeling, and physics-based synthetic signature modeling to develop in real-time a predicted target image based on imagery sensed by another sensor viewing, or having recently viewed, the same target. It is expected that the real-time algorithms will be integrated with the DARPA video verification of identity (VIVID) into a man-portable “transit case” processor.

  PHASE I: Determine the feasibility of proposed techniques for matching target and background imagery acquired from two different sensors of similar type e.g., both color TV cameras or both infrared. Investigate issues of range, perspective, differences in lighting (e.g., shadow effects), and atmospheric effects. It should be assumed that the perspective views of the targets are known from both the UAV and the manned aircraft. For example, assume it is known what the range and angles are relative to the reference frame of the vehicle being viewed. The following issues need to be addressed: 1. Effect of uncertainty in the range and perspective angle knowledge. 2. When and to what degree warping can be used to bring images of different perspective into correspondence without producing artifacts that will significantly degrade human visual correlation. 3. Use of synthetic target image generation techniques such as SPIRITS or ASGARD 4. Potential differences in sensor spectral bands. For example, the UAS sensor is uncooled long wave infrared and the manned aircraft sensor is medium wave infrared. 5. Atmospheric effects such as spectral transmission. 6. Latency between UAV imagery and weapon platform imagery when viewing moving targets. 7. Advantage of attempting to correct for visual correspondence. 8. Evaluate the confirmatory ID (CID) algorithms of DARPA Video Verification of Identity (VIVID) processing vis-à-vis providing the UAS reference imagery.
  PHASE II: Develop real-time algorithms for remapping or rendering imagery from a UAS to correlate to that projected to be observed from a targeting pod such as the Advanced Targeting FLIR (ATFLIR) or the Litening II pod. Integrate the algorithms with the DARPA VIVID algorithms into a man-portable “transit case” processor.

  PHASE III: Demonstrate the transit case Variable Remapping Of Airborne Imagery (VRAI)/VIVID processor with an appropriate UAS and F-18 with ATFLIR. PRIVATE SECTOR COMMERCIAL POTENTIAL/

  DUAL-USE APPLICATIONS: Most any unmanned or manned airborne system working to coordinate visual imagery with ground personnel or other manned aircraft will benefit from image remapping. Applications such as law enforcement, fire rescue, boarder patrol, agricultural, fish and wildlife tracking, high tension line tracing, and geological survey all have obvious needs for oriented visual signal interfaces.

  References:  1. Strat, Thomas M. and Hollan, Lois C., eds., Video Verification of Identity (VIVID): Automated Video Processing for Unmanned Aircraft – A compilation of scientific papers and technical reports that summarizes the accomplishments of the DARPA VIVID program, Phase 1, Approved for Public Release 2. Merchant, John, "Exact area registration of different views of a common object scene", Optical Engineering, 20(3), pp. 424-436 (May/June 1981) 3. Sali, E. and Ullman, S., “Recognizing novel 3-D objects under new illumination and viewing position using a small number of example views or even a single view,” Computer Vision, 1998. Sixth International Conference on, 4-7 Jan. 1998 Page(s): 153 - 161

Keywords:  Image Processing; Model-Based Rendering; Image-Based Rendering; Target Recognition; Target Identification; Video Correlation

Additional Information, Corrections, References, etc:
Ref #1: 01_paper.pdf
Ref #1: 06_paper.pdf
Ref #1: 07_paper.pdf
Ref #1: copyright.pdf
Ref #1: 08_paper.pdf
Ref #1: 09_paper.pdf
Ref #1: 10_paper.pdf
Ref #1: 01_paper.pdf
Ref #1: 06_paper.pdf
Ref #1: 07_paper.pdf
Ref #1: copyright.pdf
Ref #1: 08_paper.pdf
Ref #1: 09_paper.pdf
Ref #1: 10_paper.pdf

Questions and Answers:
Q: 1. Can you please clarify what is meant by issues #7 and #8? Specifically with #7: what is "visual correspondence"? Does this pertain to the differences between UAS VIS/color VIS/mono and/or IR?

2. Specifically with #8: If I understand the intended architecture/Conops, the VIVID CID algorithms are meant for UAS image-tracking, which is presumed a "black box" for this project, whereas VRAI isn't intended to actually track the remapped image from the separate weapons platform (which may have its own CID). Or is the ATFLIR to (via telelink?) employ the transit case processor for acquisition itself?

3. Can you clarify the extent of expected Phase II effort pertaining to VIVID algorithms per se? Specifically, are the VIVID algorithms already presumed to be in real-time execution form, or is effort expected to render them as so?

4. Based on open-source browsing, it is unclear what constitutes the "VIVID algorithms". Under VIVID Phase 1, numerous awardees contributed. Is it now the case that BAE/Alphatech is the architect? The point of this query is to gauge how much familiarity is required with the VIVID software at a deep implementation-software level, versus whether an offeror can reasonably expect to interface via a well-established API.
A: 1. "Visual correspondance" is the processing that brings the image acquired by one sensor at one perspective into visual similarity to that acquired by another sensor that views the target from another perspective. Correspondance not only refers to the geometry but also to diffences due to spectral differences in the sensors. It is to be
assumed however that the two cameras are of a similar nature, e.g., both color TV. However, thermal cameras may operate in different wavebands, e.g., on in the 8 - 12 micrometer band and the other in the 3 - 5 micrometer band.

2. The VIVID CID algorithms retain a visual memory or model of what a target and nearby confuser vehicles have looked like from the various viewed perspectives while they have been in track by the UAS or other "spotter" camera system. In addition, the CID algorithms attempt some degree of remapping of the stored target/confuser images for the purpose of determining a degree of match between "query" images of the real target and surrounding "confuser" vehicles and the "learning" imagery previously acquired while the target and confuser vehicles have been tracked. The question raised in issue #8 is whether and how such algorithms may actually provide a source of information useful to the VRAI processing.

3. The VIVID algorithms will be in real-time execution form. If in Phase 1 it is determined that the VIVID CID processing should be a part of a VRAI process then the relevant VIVID CID algorithms and software will be provided to support their integration into a VRAI processor.

4. BAE is the architect of the Multitarget Tracking (MTT) and Sensor Resource Manager portions of the overall VIVID algorithm suite. SAIC and Sarnoff were the competing contractors for the CID portion of VIVID. SAIC was chosen to continue the development and integration of real-time CID algoritms into a real-time VIVID processor. In the absence of specific algorithm descriptions proposers should presume that the VIVID
CID algorithms retain a visual history of distinct views of a target previously seen while the vehicle has been in track.
Q: 1. Can you please clarify what is meant by issues #7 and #8? Specifically with #7: what is "visual correspondence"? Does this pertain to the differences between UAS VIS/color VIS/mono and/or IR?

2. Specifically with #8: If I understand the intended architecture/Conops, the VIVID CID algorithms are meant for UAS image-tracking, which is presumed a "black box" for this project, whereas VRAI isn't intended to actually track the remapped image from the separate weapons platform (which may have its own CID). Or is the ATFLIR to (via telelink?) employ the transit case processor for acquisition itself?

3. Can you clarify the extent of expected Phase II effort pertaining to VIVID algorithms per se? Specifically, are the VIVID algorithms already presumed to be in real-time execution form, or is effort expected to render them as so?

4. Based on open-source browsing, it is unclear what constitutes the "VIVID algorithms". Under VIVID Phase 1, numerous awardees contributed. Is it now the case that BAE/Alphatech is the architect? The point of this query is to gauge how much familiarity is required with the VIVID software at a deep implementation-software level, versus whether an offeror can reasonably expect to interface via a well-established API.
A: 1. "Visual correspondance" is the processing that brings the image acquired by one sensor at one perspective into visual similarity to that acquired by another sensor that views the target from another perspective. Correspondance not only refers to the geometry but also to diffences due to spectral differences in the sensors. It is to be
assumed however that the two cameras are of a similar nature, e.g., both color TV. However, thermal cameras may operate in different wavebands, e.g., on in the 8 - 12 micrometer band and the other in the 3 - 5 micrometer band.

2. The VIVID CID algorithms retain a visual memory or model of what a target and nearby confuser vehicles have looked like from the various viewed perspectives while they have been in track by the UAS or other "spotter" camera system. In addition, the CID algorithms attempt some degree of remapping of the stored target/confuser images for the purpose of determining a degree of match between "query" images of the real target and surrounding "confuser" vehicles and the "learning" imagery previously acquired while the target and confuser vehicles have been tracked. The question raised in issue #8 is whether and how such algorithms may actually provide a source of information useful to the VRAI processing.

3. The VIVID algorithms will be in real-time execution form. If in Phase 1 it is determined that the VIVID CID processing should be a part of a VRAI process then the relevant VIVID CID algorithms and software will be provided to support their integration into a VRAI processor.

4. BAE is the architect of the Multitarget Tracking (MTT) and Sensor Resource Manager portions of the overall VIVID algorithm suite. SAIC and Sarnoff were the competing contractors for the CID portion of VIVID. SAIC was chosen to continue the development and integration of real-time CID algoritms into a real-time VIVID processor. In the absence of specific algorithm descriptions proposers should presume that the VIVID
CID algorithms retain a visual history of distinct views of a target previously seen while the vehicle has been in track.
Q: 1. Regarding the statement, "It should be assumed that the perspective views of the targets are known from both the UAV and the manned aircraft," can you please clarify what is meant by "perspective views"? Specifically, does "perspective views" mean the 2D images seen from those aircraft, or the relative 3D positions and orientations of those aircraft relative to the target, or both of the above, or neither?

2. Is the primary emphasis of the task to perform matching between imagery from two different platforms, or to extrapolate synthetic views from imagery from one platform?
A: 1. "Perspective View" relates to having knowledge of the relative positions and orientations of the target and the aircraft imaging the target. This knowledge would aid in reconstructing a 2-D image of the target from a new perspective view or synthesizing the 2-D image for the current based on those acquired from other views of the target.

2. The intent is to synthesize or remap imagery obtained from one platform to visually correspond to the imagery of the target as seen from a second platform viewing the target from a different perspective view. The intent is to aid the pilot is making a manual visual correspondence between the imagery acquired by the "spotter" aircraft to that being obtained by his own sensor.
Q: 1. Regarding the statement, "It should be assumed that the perspective views of the targets are known from both the UAV and the manned aircraft," can you please clarify what is meant by "perspective views"? Specifically, does "perspective views" mean the 2D images seen from those aircraft, or the relative 3D positions and orientations of those aircraft relative to the target, or both of the above, or neither?

2. Is the primary emphasis of the task to perform matching between imagery from two different platforms, or to extrapolate synthetic views from imagery from one platform?
A: 1. "Perspective View" relates to having knowledge of the relative positions and orientations of the target and the aircraft imaging the target. This knowledge would aid in reconstructing a 2-D image of the target from a new perspective view or synthesizing the 2-D image for the current based on those acquired from other views of the target.

2. The intent is to synthesize or remap imagery obtained from one platform to visually correspond to the imagery of the target as seen from a second platform viewing the target from a different perspective view. The intent is to aid the pilot is making a manual visual correspondence between the imagery acquired by the "spotter" aircraft to that being obtained by his own sensor.

Record: of