Deep Neural Networks, especially Convolutional Neural Networks (CNN), allows computational models that are composed of multiple processing layers to learn representations of data with multiple levels of abstraction. These methods have dramatically improved the state-of-the-arts in visual object recognition, object detection, text recognition and many other domains such as drug discovery and genomics.
In addition, many solid papers have been published in this topic, and some high quality open source CNN software packages have been made available. There are also well-written CNN tutorials or CNN software manuals. However, it might lack a recent and comprehensive summary about the details of how to implement an excellent deep convolutional neural networks from scratch. Thus, they collected and concluded many implementation details for DCNNs.
Here they will introduce these extensive implementation details, i.e., tricks or tips, for building and training your own deep networks.
Table of Contents:
Sec. 1: Data Augmentation
Sec. 2: Pre-Processing
Sec. 3: Initializations
- All Zero Initialization
- Initialization with Small Random Numbers
- Calibrating the Variances
- Current Recommendation
Sec. 4: During Training
- Filters and pooling size.
- Learning rate.
- Fine-tune on pre-trained models.
Sec. 5: Activation Functions
- Rectified Linear Unit
- Leaky ReLU
- Parametric ReLU
- Randomized ReLU
Sec. 6: Regularizations
- L2 regularization
- L1 regularization
- Max norm constraints
Sec. 7: Insights from Figures
- As we have known, the learning rate is very sensitive.
- Now let’s zoom in the loss curve.
- Another tip comes from the accuracy curve.
Sec. 8: Ensemble
- Same model, different initialization.
- Top models discovered during cross-validation.
- Different checkpoints of a single model.
- Some practical examples.
References & Source Links
- Career: Training | Books | Cheat Sheet | Apprenticeship | Certification | Salary Surveys | Jobs
- Knowledge: Research | Competitions | Webinars | Our Book | Members Only | Search DSC
- Buzz: Business News | Announcements | Events | RSS Feeds
- Misc: Top Links | Code Snippets | External Resources | Best Blogs | Subscribe | For Bloggers
- What statisticians think about data scientists
- Data Science Compared to 16 Analytic Disciplines
- 10 types of data scientists
- 91 job interview questions for data scientists
- 50 Questions to Test True Data Science Knowledge
- 24 Uses of Statistical Modeling
- 21 data science systems used by Amazon to operate its business
- Top 20 Big Data Experts to Follow (Includes Scoring Algorithm)
- 5 Data Science Leaders Share their Predictions for 2016 and Beyond
- 50 Articles about Hadoop and Related Topics
- 10 Modern Statistical Concepts Discovered by Data Scientists
- Top data science keywords on DSC
- 4 easy steps to becoming a data scientist
- 22 tips for better data science
- How to detect spurious correlations, and how to find the real ones
- 17 short tutorials all data scientists should read (and practice)
- High versus low-level data science
via Featured Blog Posts http://ift.tt/1gdozRJ
July 12, 2016 at 04:49AM