Alfred Yu

Caffe Deep Learning framework source code reading notes

Fri, 20 Jan 2017 00:00:00 +0000

Caffe Introduction

Caffe Tool

Basic tips for C/C++ compiler or make

Mon, 09 Jan 2017 00:00:00 +0000

Shared, Static library

Recent end-to-end object detection deep learning methods

Thu, 01 Dec 2016 00:00:00 +0000

Deep learning has really boosted the application of compute vision.

There are a few guys who are quite active in the research community.

Some fundamental questions:

1.What does the response in the feature map really mean? Means some feature being activated?

2.Is CNN recognizing the object by their shape information?

RCNN

SSP-net

Spatial Pyramid Pooling in Deep Convolutional Networks for visual Recognition (SSP-net)

For SSP-net, what does each spatial bin really contribute to the final results?

For SSP-net, why cannot it fine-tune the layers precede the spatial pyramid pooling?

Because the training is designed to train the model with the ROIs from different images, which is not efficient

Fast-RCNN

The main contribution:

Training is single-stage, using a multi-task loss.
Training can update all network layers (comparing to SSP-net).

The questions:

For bounding-box regression, do we need output 4xK values for each class?
Spatial pyramid pooling structure, in fast-RCNN only one lever of pyramid, HxW bins, while in SSP-net there are hierarchical levers of pyramids.
For fine-tuning, how can hierarchical sampling become a contribution of the paper? And what’s the detail of hierarchical sampling? and how SGD is implemeted regarding the sampling method?
For bounding-box regression, why is $t^k$ defined to specify a scal-invariant translation and log-space height/width shift relative to an object proposal.

TODO: understand the loss function defintion, and why use smooth L1 function?

Two ways of achieving scale invariant object detection:

brute-force learning: each image is processed as a pre-defined pixel size during both training and testing. The network must directly learn scale-invariant object detection from the traning data. And it’s proved by experiments that this approach is better.
image pyramids: providing approximate scale-invariance to the network through an image pyramid. Figure out implementation detail.

Why we need the distributed training? If one model need to train more than one hours, then it’s prohibitable to try many different methods or functions which may improve the model.

Another Trick: Truncated SVD for faster detection (Approximation idea).

Exact model & approximated solution vs. Approximated model &exact solution

Idea: figure out which layer is task dependent, then it need to be fine-tuned when learning the specific task. Otherwise, it can be shared among different task. Usually, the first layer is task independent (a well known fact).

Faster RCNN

The main bottleneck for the detection using fast RCNN is the region proposal. And usually the region proposal is done in CPU instead of GPU. The idea is to speed up the region proposal by making advantage of GPU.

Idea: `sharing computation of convolutions which is efficient, yet accurate method for visual recognition.

Region Proposal Network is still doing the sliding windows, but based on the convolutional layer, which shares the computation with the detection. Also notice that the anchor plays the role of pyramid.

One challenge of detection is to handle the translation-invariant issue. It’s easy to understand that the convolutional layer can be shift-invariant, but how to understand that the layer can be scale-invariant.

THe design of multi-scale anchors is a key component for sharing features without extra cost for addressing scales.

Also be careful about the selection of positive and negative samples.

a set of $k$ bounding-box regressors are learned ?

Training RPN & Fast RCNN. The basic idea is to use deep nets to extract the features which can be shared between the proposal network and detection network.

Bounding-box regression is important for classification.

Implementation detail: number of proposals , scales, ratios (anchor) , choose the negative samples.

One idea about the end-to-end learning framework is that if we have a model with large capability, then more accuracy can be obtained when feed more data into the model.

YOLO

Frame object detection as a regression problem to spatially separated bounding boxes and associated class probabilities.

The main idea for detection is how to perform bounding box and class probabilities prediction based on the feature sharing,

Encode contextual information about classes. (How to encode the global information of the image). Fast RCNN cannot.

Grid cell, and do regression on each cell to predict the probability of the class. And like DPM, it can model the size and shape of the objects.

Compare faster RCNN and YOLO.

YOLO is good at proposal.

YOLO struggles with small objects compared to its closest competitors.

Implementation: Pr(Class

Object) only predict one set of class probabilities per grid cell, regardless of the number of boxes B, why?

To summary: the idea of the yolo is to sharing the 7 x 7 image classification problem, and cast the classification problem to be a regression problem. And

SSD

The first deep network based object detector which does not resample pixels or features for bounding box hypotheses and is as accurate as approaches that do.

Using a small convolutional filter to predict object categories and offsets in bounding box locations, using separate predictors (filters) for different aspect ratio detections, and applying these filters to multiple feature maps from the later stages of a network in order to perform detection at multiple scales.

The core of SSD is predicting category scores and box offsets for a fixed set of default bounding boxes using small convolutional filters applied to feature maps.
To achieve high detection accuracy we produce predictions of different scales from feature maps of different scales, and explicitly separate predictions by aspect ratio.

The SSD approach is based on a feed-forward convolutional network that produces a fixed-size collection of bouding boxes and scores for the presence of object class instances in those boxes.

Hierarchical structure of the feature map for different scale and ratio.

Data augmentation is crucial.

SSD is very sensitive to the bounding box size. In other words, it has much worse performance on smaller objects than bigger objects. This is not surprising because those small objects may not even have any information at the very top laryers.

Feature Pyramid Networks for object detection

The construction of our pyramid involves a bottom-up pathway, a top-down pathway, and lateral connections

Is it true that the hierarchy of the network gives the power of different scale?

Understand Jacobian Factor

Tue, 01 Nov 2016 00:00:00 +0000

Recently, I am going to rewiew the the book “Pattern Recognition and Machine Learning”. It is the best and most comprehensive book about machine learning. While reading the book, I will try to review all the foundamental concepts, mainly in the probability, linear algebra and optimization. The hope is that it can be served as a guildline for others who are interested in exploring machine learning area.

Jacobian Factor

Jacobian

Object Detection

Mon, 01 Aug 2016 00:00:00 +0000

#Object Proposal

Object tracking

Fri, 01 Jul 2016 00:00:00 +0000

For automatic/intelligent industry application, object detection and object tracking are critical step. Take a step further,

##Scale-invariant feature transform (SIFT)

SIFT feature descriptor is invariant to uniform scaling, orientation and partially invariant to affine distortion and illumination changes.

Scale spaces

The first step is to create a scale space. Take the original image and generate progrssively blurred out images.

Learning CMake

Wed, 01 Jun 2016 00:00:00 +0000

Object Detection with Discriminatively Parted Based Models

Mon, 01 Feb 2016 00:00:00 +0000

In this article, I will discuss about three papers, which are start-of-the-art of object detection. They are 1. Cascade 2. Histogram of oriented gradients 3. Deformable part model

Cascade

The paper of the paper is very simple. There are mainly two steps to do object detection. 1.

Histogram of oriented gradients

Deformable part model

Object Detection Basic 1

Tue, 26 Jan 2016 00:00:00 +0000

In computer vision area, object detection is the first step before we can do the object recognization. In this article, I will try to summarize my first knowledge about object detection techniques.

State-of-the-art

The cascade is the most successful detection technique first appeared in 2001. The idea is quite simple. The raw pixel is not used as the features, but instead cascade is building some features based on the rectange area. Physically, it defines the feature as the first two order of gradients. The feature dimension can be very large. They applied AdaBoost to reduce the feature dimension. After that, they apply the cascade algorithm to speed up the detection procedure.

Histogram of oriented gradient

In HOG, they have designed a new feature sets, which is based on the histogram of oriented gradient. Then apply the SVM to train the data.

DPM

DPM is the most recent one. There are many details inside. The basic idea is to encode the object information in two levels of hierarchy. The root filter is a coarse feature map. The part model is twice the spatial resolution relative to the features captured by the root.

Basic model

$\begin{align} f_\beta(x) & = \max_z \beta \Phi(x,z) \\ \beta &= (F_0, \ldots, F_n, d_1, \ldots, d_n, b) \end{align}$

where $\beta$ is the concatenation of the root filter, the part filters and deformation cost weights. $z$ is the specification of the object, and $z = (P_0, \ldots, p_n)$ and $p_i = (x_i, y_i, l_i)$ . Here we denote the model using the tuple $(F_0, P_1, \ldots, P_n)$ where $F_0$ is the root filter, and each model is defined by 3-tuple $(F_i, v_i, d_i)$ , $F_i$ is the filter for the $i$ -th part, $v_i$ is anchor position, and $d_i$ is a 4-dimensional coefficients of the quadratic function defining the deformation cost model. So the objective function is

$\text{score}(p_0, \ldots, p_n) = \sum_{i=0}^n F_i \phi(H,p_i) - \sum_{i=0}^n d_i \phi_d(d_{x_i}, d_{y_i}) + b$

wphere p $(d x_i , d y_i) = (x_i, y_i) - (2(x_0,y_0) + v_i)$

is the displacement of the $i$ -th part relative to its anchor position and

$\phi_d(dx , dy) = (dx, dy, dx^2, dy^2)$

are deformation features.

Matching

To detect the objects in the image, we compute the score for each root location according to best placement of parts,

$score(p_0) = \max_{p_1, \ldots, p_n} score(p_0, \ldots, p_n)$

Distance transforms (dynamic programming) can be used to compute the score efficiently. The computational procedure can be briefed as

$\begin{align} R_{i,l}(x,y) &= F_i \phi(H, (x,y,l)) \\ D_{i,l}(x,y) &= \max_{dx,dy} (R_{i,l}(x +dx, y + dy) - d_i \phi_d(dx,dy)) \\ score(x_0,y_0, l_0) &= R_{0,l_0} + \sum_{i=1}^n D_{i,l_0 - \lambda} ( 2 (x_0, y_0) + v_i) + b \end{align}$

Latent SVM

The model can be trained using SVM, which is

$L_D(\beta) = \frac{1}{2} \| \beta\|^2 + C \sum_{i=1}^n \max(0, 1- y_i f_{\beta}(x_i))$

We should not that the object function is not convex. However it’s semi-convex since it’s convex for negative examples.

If there is a single possible latent value for each positive example, then $f_\beta(x_i)$ is linear for a positive example and the loss due to each positive is convex, then the loss function is convex.

Optimization

Let $Z_p$ specify a latent value for each positive example, we define $L_D(\beta, Z_p) = L_{D(Z_p)}(\beta)$ ,

$L_D(\beta) = \min_{Z_p} L_D(\beta, Z_p)$

Algorithms

relabel positive examples: Optimizae $L_D(\beta, Z_p)$ over $Z_p$ by $z_i = \arg\max_z \beta \Phi(x_i,z)$
Optimize beta: Optimize $L_D(\beta, Z_p)$ .

Data-mining hard examples

There are enormous amounts of negative examples in the database. Hard-example can be used to speed up the training. The idea is that we can reduce the negative examples to hard example, but keep the optimal solution unchanged.

A tutorial for AdaBoost

Tue, 26 Jan 2016 00:00:00 +0000

Tutorial

AdaBoost is based on crowdsourcing idea to build a strong classifier. A pool of weak classifiers is provided, and in each iteration, the algorithm will pick one weak classifier from the pool.