Traditional video surveillance uses manual video surveillance to discover potential security risks or abnormal states, or for post-event analysis. This application has its inherent shortcomings, and it is difficult to realize real-time safety monitoring and detection management. The monitoring system with intelligent analysis function actively collects and analyzes data by distinguishing the shape, action and other characteristics of the monitored object, and performs actions such as alarm, recording and analysis according to preset conditions. Intelligent monitoring system can run on the server or embedded system based on DSP, and the latter has gradually become the mainstream.
The application of intelligent video can be divided into three aspects: security, human behavior detection and intelligent transportation. Among them, security application is widely regarded as the most potential market, which includes the following applications: intrusion detection, which can automatically detect the characteristics of motion behavior in video images; Cargo moving detection can automatically detect cargo moving events-when the goods at a specific location in the defense area are taken away or moved, it will give an alarm; Legacy detection, which can automatically detect abandoned items-automatically alarm when placing or abandoning items in a defense zone; Intelligent tracking enables the camera to drive its own pan/tilt and zoom lens independently by PTZ. The applications of human behavior detection include off-duty detection (which can automatically detect the position of sentry personnel) and wandering detection (which can detect human wandering in important areas). Intelligent transportation applications include: detecting illegally parked vehicles and giving an alarm when vehicles illegally stay in the defense zone; Vehicle retrograde detection, timely identification of retrograde vehicles.
With the gradual improvement of accuracy and reliability and the decline of product cost, intelligent video has been applied in more and more occasions, which can replace some security equipment, reduce the work intensity of security personnel, improve work efficiency and reduce management costs. In fact, the application potential of intelligent video is great. With the maturity of technology, the application field of intelligent video technology is expanding rapidly. These applications mainly include the above-mentioned security, transportation, retail and service industries, such as demography, face recognition, crowd control, attention control and traffic flow control.
The demand for real-time video monitoring is growing rapidly, especially for real-time security monitoring applications. The function of real-time detection of security risks or abnormal behaviors of targets is becoming more and more important. Driven by this growing demand, intelligent video surveillance system products are becoming a new hot spot in video surveillance applications. Especially with the progress of semiconductor technology, such as the introduction of advanced embedded solution platform represented by Blackfin convergent processor, intelligent video analysis equipment with high cost performance and high practicability is constantly introduced, and plays an extremely important role in some key applications.
Figure 1: Schematic diagram of traditional video surveillance application.
Intelligent video application design strategy
The choice of hardware platform scheme often determines the cost, performance, availability of development tools and methods, and the feasibility of continuous upgrading in the future, so the choice of scheme platform is very important. The uniqueness of intelligent video application requires a comprehensive balance in the choice of hardware platform. The requirements of networking and intelligent analysis of video surveillance system, as well as the limitation of large-scale engineering installation on cost, volume and power consumption, non-standardized intelligent video analysis method and almost customized scheme optimization method, make the integrated processor scheme platform which combines the advantages of MCU and DSP have more obvious advantages in software design flexibility and powerful processing ability. This paper will combine the characteristics of the unique Blackfin convergent DSP processor of analog devices, and analyze the main design technical points in intelligent video design.
First, the choice of hardware platform
Customization is very important. There are many factors that restrict the intelligent application of video surveillance system: first, the video algorithm of intelligent surveillance is complex and difficult to standardize, and the video analysis software of each system provider has its own unique algorithm, resulting in no unified standard for products on the market; Secondly, the video surveillance system has complex application scenarios, diversified user needs and many customization needs. Therefore, video analysis schemes usually need to be optimized according to the application characteristics and needs of customers, and the algorithms used are very different. In addition, due to the high complexity of intelligent video applications, higher requirements are put forward for the processing capacity of the scheme. The traditional video monitoring scheme of MCU+ASIC is difficult to achieve various personalized designs and high computing power requirements. Even if the dual-chip scheme of ordinary DSP+MCU is selected, it is usually difficult to meet the complex computing requirements of intelligent video surveillance applications, and it is necessary to increase the coprocessor. This complex solution is not enough in BOM cost, power consumption and development difficulty. Blackfin processor gives full play to the advantages of MCU+DSP fusion architecture, and meets the system control and high-intensity computing requirements of intelligent video applications, especially the high-performance dual-core architecture represented by BF56 1 has become the preferred platform for intelligent video applications.
The scalability of the scheme is also a factor to be considered. The application of intelligent video analysis needs to be optimized according to the application environment and application purpose, and different customers may have different needs in other aspects. For example, some leading digital video surveillance schemes have already supported H.264 basic classes @Level3.0 and MPEG-4D 1+CIF, and may be extended to support H.264 D 1+CIF in the future. With the wide application of intelligent video analysis, such as IP camera, wireless video surveillance, intelligent transportation system and so on. Different applications may have different requirements for various interface functions, communication standards and user interfaces. And the hardware platform scheme is very important for the flexibility and expansibility of various requirements. At the same time, as mentioned above, the development of intelligent video analysis technology is only a few years. With the continuous maturity of technology and the introduction and improvement of some related standards, the upgrade of products is very important, which is not only a concern of developers, but also an important feature of end customers. Blackfin DSP has unique advantages in algorithm parallel processing, especially ADSP-BF56 1 adopts dual DSP cores, which can realize very complex intelligent video processing algorithm.
Optimization characteristics of video applications. Although some schemes have strong processing ability and expansibility, it is also worth paying attention to whether the schemes are optimized for video applications, because this is directly related to the available software and hardware design resources of design engineers, the difficulty of system design and the achievable performance. Taking Blackfin processor as an example, Blackfin is specially optimized for digital and media processing with high intensity and high data rate: Blackfin's dozens of DMA channels and flexibly configurable cache meet the requirements of video surveillance system for large computation and high data throughput; ADI has specially developed fully optimized audio and video codecs and provided them to large customers free of charge. Blackfin integrates many hardware drivers for video applications, including WiFi drivers and audio/video codec drivers. Blackfin's four video arithmetic operation units and video pixel instruction set greatly accelerate the video operation speed; In some basic operations of intelligent video analysis, such as histogram statistics, median operation, Sobel operation and expansion operation in morphology, Blackfin's MIN and MAX instructions can be used to eliminate conditional jumps and save processor cycles. Not only that, Blackfin also supports vector operations of 13 kinds of non-video data. By reasonably designing the data structure, Blackfin's characteristic instructions can be used to make the intelligent video analysis algorithm faster in many links such as foreground and background separation, threshold calculation and update. These instructions are very effective in themselves, and most of them can be executed in parallel, thus doubling Blackfin's processing power.
Low power consumption and stability are very important. Considering that intelligent video surveillance equipment usually runs 24 hours a day, 7 days a week, stability and power consumption are more important. In terms of low power consumption, Blackfin processor adopts a variety of energy-saving technologies: based on the design of gated clock core, it can selectively cut off the power supply of functional units one by one according to instructions; Support multiple power-off modes during the minimum CPU action; Blackfin processor supports independent dynamic power management circuit, which can independently control the working frequency and voltage to meet the performance requirements of the algorithm being executed. Most Blackfin processors provide on-chip core voltage regulator circuits, which can work as low as 0.8V Blackfin's unique aggregation processing architecture and 90nm technology have laid the foundation for its leading low-power processing. Due to the high processing capacity, the system scheme based on Blackfin platform can reduce the number of main chips, and rich functions and interfaces can meet the needs of various peripherals and function expansion, reducing the number of components, thus ensuring higher stability and reliability. Blackfin DSP is the best DSP with low power consumption and stability at present.
What embedded operating systems are supported? Intelligent video analysis is usually a network-based application and must be supported by the operating system. Therefore, it is very important to choose a solution with a wide range of embedded system support capabilities, so as to ensure that the hardware platform does not need to be replaced when replacing the operating system in future products, and to ensure the continuous availability of research and development results. At present, there are many embedded operating systems available, each with its own advantages, and the support ability of hardware platform scheme for these operating systems is one of the key points to be considered in scheme selection. For example, Blackfin processor can support mainstream operating systems, including more than ten embedded operating systems such as uCLinux, ThreadX, Nucleus and uCOS-II. Customers can choose their own familiar or more cost-effective software architecture foundation according to their own requirements.
Figure 2: Block diagram of intelligent monitoring terminal based on BF56 1.
Second, develop tools and available resources.
Intelligent video surveillance equipment is a complex system, involving complex software and hardware design, man-machine interface, communication connection and so on. The system design is difficult. Therefore, whether the selected hardware platform scheme can provide a complete development tool suite, necessary software modules, mature reference design and system design support, and whether there is a complete design ecosystem is very critical for whether the system design can be completed on schedule with high quality. In fact, not all platform solution providers can provide such support.
Taking Blackfin series processors as an example, the hardware platform using Blackfin processors has been widely adopted by a large number of equipment companies around the world, from general DVR, IP camera, digital video surveillance to intelligent video surveillance. Blackfin processor is favored by many enterprises, and the support of complete development tools and reference design is one of the important reasons for its popularity. ADI provides industry-class tools, starter kits and support, including the famous ADI CROSSCORE? Software and hardware tools, including the award-winning VisualDSP++? Integrated Development and Debugging Environment (IDDE), Emulator and EZ-KIT Lite? Evaluation version of the hardware.
In order to improve development efficiency and reduce development difficulty, we should try our best to develop on existing resources, such as open routines, and ADI has provided a wealth of routines and materials for this purpose. For example, ADI provides a free "Image Toolbox" image processing function library software package, which is specially optimized for mathematical functions commonly used in image processing applications for customers to call when developing applications. ADI also provides complete reference designs, evaluation boards, development tools, algorithm IP and application modules developed by local partners, and complete turnkey solutions, including software and hardware, provided by third-party partners. At present, the application of Blackfin processor in the field of video surveillance has had many third-party partners with rich engineering experience in China, and a perfect ecosystem has been established.
Take the complete IP monitoring and machine vision camera reference design based on Blackfin BF526C released by ADI in March this year as an example. The reference design provides powerful video and audio processing capabilities on a single converged processor, provides a unified software development environment for engineers, and can realize faster system debugging and deployment and lower system cost. The processor provides integrated audio codec, streaming video and IP protocol, on-chip DRAM memory and interface for 10/ 100 Ethernet, USB and SD storage and local RS-232 port. This fully programmable solution can meet various video compression standards, such as H.264 and MPEG4-4, and support the coding of audio G.729 standard. It supports two-way voice communication from the control center to the camera, as well as lens translation, tilt and stretching using Pelo-P or Pelo-D protocols. The reference design also provides a daughter card with dual-core BF56 1 processor, which enables the system to achieve higher video resolution and provide advanced video analysis functions, such as motion detection and tracking.
Revealing the Application Scheme —— Yiwei Oriental Intelligent Network Camera
Emvideo is a solution provider of professional intelligent security products and an authorized third-party partner of ADI. At present, Evian Oriental has many products based on the hardware platform of ADI Blackfin processor, among which the overall solution of "software+hardware" turnkey WiFi wireless video surveillance is based on the dual-processor architecture of BF 536+BF 56 1, and the hardware structure diagram of this solution is shown in Figure 3.
Among them, BF536 processor, as the main processor, is not only responsible for some basic management and control such as audio coding, remote control and user interaction control, but also for the operation of embedded operating system uClinux and advanced intelligent video analysis functions, which can complete a variety of intelligent video analysis such as security, human behavior and intelligent transportation. Dual-core BF56 1 as a coprocessor is responsible for video coding algorithm. The powerful video processing capability enables the scheme to support the dual-code stream of H.264 basic class @Level3.0 and MPEG-4D 1+CIF, and it is likely to be extended to support the dual-code stream of H.264 D 1+CIF in the future. The two processors can communicate through a high-speed synchronous serial interface. The video signal first enters the BF56 1 processor, and the encoded code stream is collected and sent to the BF536 processor, and then sent to the client for decoding and display through the network.
Figure 3: Hardware structure diagram of the solution using Blackfin BF536 and BF56 1.
The scheme adopts advanced background modeling method, which can effectively overcome the interference of background such as light change, leaf swing and water ripple on foreground target analysis and realize accurate foreground detection. At the same time, the unique optimization calculation and amplification are adopted in target tracking, and the accuracy of intrusion detection (including regional early warning and trip detection) is over 90%. These are the powerful processing capabilities of the hardware architecture based on BF 536+BF 56 1 dual processors.
The intelligent video analysis function of this scheme is independently developed by Yiwei Oriental Company. The unique algorithm and rich experience in the development of intelligent video analysis technology ensure the realization of customers' intelligent identification application requirements and provide customers with perfect services including software upgrade. Because there is no requirement standard and test standard for intelligent video recognition applications at present, video analysis schemes usually need to be optimized according to the application characteristics and requirements of customers. For example, some users belong to the subway system, and the function they need is to detect whether they have crossed the waiting yellow line, whether the crowd density is too large, and whether there are suspicious remnants. Some users belong to the banking system, and what they need is intelligent monitoring of ATM machines, such as analyzing whether fake keyboards and card swallowers are installed, whether ATM machines are violent, and whether there are criminals' faces. With this scheme, customers can easily adjust the algorithm according to the needs of users. Intelligent video processing requires the powerful processing power of the chip, and many algorithms need to be processed in parallel. Blackfin DSP has unique advantages in algorithm parallel processing, especially the dual DSP cores of ADSP-BF56 1 can realize complex intelligent video processing algorithms. This is difficult to be realized by the traditional MCU+ASIC or general DSP scheme.
The software and hardware of the scheme have passed the application verification and have been produced by many customers. The target application will mainly be users in the government administrative efficiency supervision, education system and other industries.