Initial commit

Author: s9xie

.gitignore

@@ -0,0 +1,4 @@
+gen/
+libnccl.so
+model_best.t7
+checkpoints

CONTRIBUTING.md

@@ -0,0 +1,28 @@
+# Contributing to ResNeXt
+We want to make contributing to this project as easy and transparent as
+possible.
+
+
+## Pull Requests
+We actively welcome your pull requests.
+
+1. Fork the repo and create your branch from `master`.
+2. If you haven't already, complete the Contributor License Agreement ("CLA").
+
+## Contributor License Agreement ("CLA")
+In order to accept your pull request, we need you to submit a CLA. You only need
+to do this once to work on any of Facebook's open source projects.
+
+Complete your CLA here: <https://code.facebook.com/cla>
+
+## Issues
+We use GitHub issues to track public bugs. Please ensure your description is
+clear and has sufficient instructions to be able to reproduce the issue.
+
+## Coding Style  
+* 3 spaces for indentation rather than tabs
+* 80 character line length
+
+## License
+By contributing to ResNeXt, you agree that your contributions will be licensed
+under its [BSD license](https://github.com/facebookresearch/ResNeXt/blob/master/LICENSE).

LICENSE

@@ -0,0 +1,30 @@
+BSD License
+
+For ResNeXt software
+
+Copyright (c) 2017, Facebook, Inc. All rights reserved.
+
+Redistribution and use in source and binary forms, with or without modification,
+are permitted provided that the following conditions are met:
+
+ * Redistributions of source code must retain the above copyright notice, this
+   list of conditions and the following disclaimer.
+
+ * Redistributions in binary form must reproduce the above copyright notice,
+   this list of conditions and the following disclaimer in the documentation
+   and/or other materials provided with the distribution.
+
+ * Neither the name Facebook nor the names of its contributors may be used to
+   endorse or promote products derived from this software without specific
+   prior written permission.
+
+THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND
+ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED
+WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE
+DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE FOR
+ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES
+(INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES;
+LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON
+ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
+(INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS
+SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.

PATENTS

@@ -0,0 +1,33 @@
+Additional Grant of Patent Rights Version 2
+
+"Software" means the ResNeXt software distributed by Facebook, Inc.
+
+Facebook, Inc. ("Facebook") hereby grants to each recipient of the Software
+("you") a perpetual, worldwide, royalty-free, non-exclusive, irrevocable
+(subject to the termination provision below) license under any Necessary
+Claims, to make, have made, use, sell, offer to sell, import, and otherwise
+transfer the Software. For avoidance of doubt, no license is granted under
+Facebook’s rights in any patent claims that are infringed by (i) modifications
+to the Software made by you or any third party or (ii) the Software in
+combination with any software or other technology.
+
+The license granted hereunder will terminate, automatically and without notice,
+if you (or any of your subsidiaries, corporate affiliates or agents) initiate
+directly or indirectly, or take a direct financial interest in, any Patent
+Assertion: (i) against Facebook or any of its subsidiaries or corporate
+affiliates, (ii) against any party if such Patent Assertion arises in whole or
+in part from any software, technology, product or service of Facebook or any of
+its subsidiaries or corporate affiliates, or (iii) against any party relating
+to the Software. Notwithstanding the foregoing, if Facebook or any of its
+subsidiaries or corporate affiliates files a lawsuit alleging patent
+infringement against you in the first instance, and you respond by filing a
+patent infringement counterclaim in that lawsuit against that party that is
+unrelated to the Software, the license granted hereunder will not terminate
+under section (i) of this paragraph due to such counterclaim.
+
+A "Necessary Claim" is a claim of a patent owned by Facebook that is
+necessarily infringed by the Software standing alone.
+
+A "Patent Assertion" is any lawsuit or other action alleging direct, indirect,
+or contributory infringement or inducement to infringe any patent, including a
+cross-claim or counterclaim.

README.md

@@ -0,0 +1,74 @@
+# Introduction
+This repository contains a [Torch](http://torch.ch) implementation for both the [ResNeXt](https://arxiv.org/abs/1611.05431) algorithm for image classification. The code is based on [fb.resnet.torch] (https://github.com/facebook/fb.resnet.torch).
+
+[ResNeXt](https://arxiv.org/abs/1611.05431) is a simple, highly modularized network architecture for image classification. Our network is constructed by repeating a building block that aggregates a set of transformations with the same topology. Our simple design results in a homogeneous, multi-branch architecture that has only a few hyper-parameters to set. This strategy exposes a new dimension, which we call “cardinality” (the size of the set of transformations), as an essential factor in addition to the dimensions of depth and width.
+
+
+![teaser](http://vcl.ucsd.edu/~sxie/teaser.png)
+##### Figure: Training curves on ImageNet-1K. (Left): ResNet/ResNeXt-50 with the same complexity (~4.1 billion FLOPs, ~25 million parameters); (Right): ResNet/ResNeXt-101 with the same complexity (~7.8 billion FLOPs, ~44 million parameters).
+-----
+
+If you use ResNeXt in your research, please cite the paper:
+```
+@article{Xie2016,
+  title={Aggregated Residual Transformations for Deep Neural Networks},
+  author={Saining Xie and Ross Girshick and Piotr Dollár and Zhuowen Tu and Kaiming He},
+  journal={arXiv preprint arXiv:1611.05431},
+  year={2016}
+}
+```
+
+# Requirements and Dependencies
+See the fb.resnet.torch [installation instructions](https://github.com/facebook/fb.resnet.torch/blob/master/INSTALL.md) for a step-by-step guide.
+- Install [Torch](http://torch.ch/docs/getting-started.html) on a machine with CUDA GPU
+- Install [cuDNN v4 or v5](https://developer.nvidia.com/cudnn) and the Torch [cuDNN bindings](https://github.com/soumith/cudnn.torch/tree/R4)
+- Download the [ImageNet](http://image-net.org/download-images) dataset and [move validation images](https://github.com/facebook/fb.resnet.torch/blob/master/INSTALL.md#download-the-imagenet-dataset) to labeled subfolders
+
+## Training
+
+Please follow [fb.resnet.torch] (https://github.com/facebook/fb.resnet.torch) for the general usage of the code, including [how](https://github.com/facebook/fb.resnet.torch/tree/master/pretrained) to use pretrained ResNeXt models for your own task.
+
+There are two new hyperparameters need to be specified to determine the bottleneck template:
+
+**-baseWidth** and **-cardinality**
+
+###1x Complexity Configurations Reference Table
+| baseWidth | cardinality |
+|---------- | ----------- |
+| 64        | 1           |
+| 40        | 2           |
+| 24        | 4           |
+| 14        | 8           |
+| 4         | 32          |
+
+
+To train ResNeXt-50 (32x4d) on 8 GPUs for ImageNet:
+```bash
+th main.lua -dataset imagenet -bottleneckType resnext_C -depth 50 -baseWidth 4 -cardinality 32 -batchSize 256 -nGPU 8 -nThreads 8 -shareGradInput true -data [imagenet-folder]
+```
+
+To reproduce CIFAR results (e.g. ResNeXt 16x64d for cifar10) on 8 GPUs:
+```bash
+th main.lua -dataset cifar10 -bottleneckType resnext_C -depth 29 -baseWidth 64 -cardinality 16 -weightDecay 5e-4 -batchSize 128 -nGPU 8 -nThreads 8 -shareGradInput true
+```
+To get comparable results using 2/4 GPUs, you should change the batch size and the corresponding learning rate:
+```bash
+th main.lua -dataset cifar10 -bottleneckType resnext_C -depth 29 -baseWidth 64 -cardinality 16 -weightDecay 5e-4 -batchSize 64 -nGPU 4 -LR 0.05 -nThreads 8 -shareGradInput true
+th main.lua -dataset cifar10 -bottleneckType resnext_C -depth 29 -baseWidth 64 -cardinality 16 -weightDecay 5e-4 -batchSize 32 -nGPU 2 -LR 0.025 -nThreads 8 -shareGradInput true
+```
+Note: CIFAR datasets will be automatically downloaded and processed for the first time. We found that better CIFAR test acurracy can be achieved using a (on 8 GPUs) batch size of 128.
+
+# ImageNet Pretrained Models
+ImageNet pretrained models are licensed under CC BY-NC 4.0.
+
+[![CC BY-NC 4.0](https://i.creativecommons.org/l/by-nc/4.0/88x31.png)](https://creativecommons.org/licenses/by-nc/4.0/)
+
+###Single-crop (224x224) validation error rate
+| Network             | GFLOPS | Top-1 Error |  Download   |
+| ------------------- | ------ | ----------- | ------------|
+| ResNet-50 (1x64d)   |  ~4.1  |  23.9        | [Original ResNet-50](https://github.com/facebook/fb.resnet.torch/tree/master/pretrained)       |
+| ResNeXt-50 (32x4d)  |  ~4.1  |  22.2        | [Download (191MB)](https://s3.amazonaws.com/resnext/imagenet_models/resnext_50_32x4d.t7)       |
+| ResNet-101 (1x64d)  |  ~7.8  |  22.0        | [Original ResNet-101](https://github.com/facebook/fb.resnet.torch/tree/master/pretrained)      |
+| ResNeXt-101 (32x4d) |  ~7.8  |  21.2        | [Download (338MB)] (https://s3.amazonaws.com/resnext/imagenet_models/resnext_101_32x4d.t7)      |
+| ResNeXt-101 (64x4d) |  ~15.6 |  20.4        | [Download (638MB)](https://s3.amazonaws.com/resnext/imagenet_models/resnext_101_64x4d.t7)       |
+

checkpoints.lua

@@ -0,0 +1,69 @@
+--
+--  Copyright (c) 2017, Facebook, Inc.
+--  All rights reserved.
+--
+--  This source code is licensed under the BSD-style license found in the
+--  LICENSE file in the root directory of this source tree. An additional grant
+--  of patent rights can be found in the PATENTS file in the same directory.
+--
+local checkpoint = {}
+
+local function deepCopy(tbl)
+   -- creates a copy of a network with new modules and the same tensors
+   local copy = {}
+   for k, v in pairs(tbl) do
+      if type(v) == 'table' then
+         copy[k] = deepCopy(v)
+      else
+         copy[k] = v
+      end
+   end
+   if torch.typename(tbl) then
+      torch.setmetatable(copy, torch.typename(tbl))
+   end
+   return copy
+end
+
+function checkpoint.latest(opt)
+   if opt.resume == 'none' then
+      return nil
+   end
+
+   local latestPath = paths.concat(opt.resume, 'latest.t7')
+   if not paths.filep(latestPath) then
+      return nil
+   end
+
+   print('=> Loading checkpoint ' .. latestPath)
+   local latest = torch.load(latestPath)
+   local optimState = torch.load(paths.concat(opt.resume, latest.optimFile))
+
+   return latest, optimState
+end
+
+function checkpoint.save(epoch, model, optimState, isBestModel, opt)
+   -- don't save the DataParallelTable for easier loading on other machines
+   if torch.type(model) == 'nn.DataParallelTable' then
+      model = model:get(1)
+   end
+
+   -- create a clean copy on the CPU without modifying the original network
+   model = deepCopy(model):float():clearState()
+
+   local modelFile = 'model_' .. epoch .. '.t7'
+   local optimFile = 'optimState_' .. epoch .. '.t7'
+
+   torch.save(paths.concat(opt.save, modelFile), model)
+   torch.save(paths.concat(opt.save, optimFile), optimState)
+   torch.save(paths.concat(opt.save, 'latest.t7'), {
+      epoch = epoch,
+      modelFile = modelFile,
+      optimFile = optimFile,
+   })
+
+   if isBestModel then
+      torch.save(paths.concat(opt.save, 'model_best.t7'), model)
+   end
+end
+
+return checkpoint

dataloader.lua

@@ -0,0 +1,127 @@
+--
+--  Copyright (c) 2017, Facebook, Inc.
+--  All rights reserved.
+--
+--  This source code is licensed under the BSD-style license found in the
+--  LICENSE file in the root directory of this source tree. An additional grant
+--  of patent rights can be found in the PATENTS file in the same directory.
+--
+--  Multi-threaded data loader
+--
+
+local datasets = require 'datasets/init'
+local Threads = require 'threads'
+Threads.serialization('threads.sharedserialize')
+
+local M = {}
+local DataLoader = torch.class('resnet.DataLoader', M)
+
+function DataLoader.create(opt)
+   -- The train and val loader
+   local loaders = {}
+
+   for i, split in ipairs{'train', 'val'} do
+      local dataset = datasets.create(opt, split)
+      loaders[i] = M.DataLoader(dataset, opt, split)
+   end
+
+   return table.unpack(loaders)
+end
+
+function DataLoader:__init(dataset, opt, split)
+   local manualSeed = opt.manualSeed
+   local function init()
+      require('datasets/' .. opt.dataset)
+   end
+   local function main(idx)
+      if manualSeed ~= 0 then
+         torch.manualSeed(manualSeed + idx)
+      end
+      torch.setnumthreads(1)
+      _G.dataset = dataset
+      _G.preprocess = dataset:preprocess()
+      return dataset:size()
+   end
+
+   local threads, sizes = Threads(opt.nThreads, init, main)
+   self.nCrops = (split == 'val' and opt.tenCrop) and 10 or 1
+   self.threads = threads
+   self.__size = sizes[1][1]
+   self.batchSize = math.floor(opt.batchSize / self.nCrops)
+   local function getCPUType(tensorType)
+      if tensorType == 'torch.CudaHalfTensor' then
+         return 'HalfTensor'
+      elseif tensorType == 'torch.CudaDoubleTensor' then
+         return 'DoubleTensor'
+      else
+         return 'FloatTensor'
+      end
+   end
+   self.cpuType = getCPUType(opt.tensorType)
+end
+
+function DataLoader:size()
+   return math.ceil(self.__size / self.batchSize)
+end
+
+function DataLoader:run()
+   local threads = self.threads
+   local size, batchSize = self.__size, self.batchSize
+   local perm = torch.randperm(size)
+
+   local idx, sample = 1, nil
+   local function enqueue()
+      while idx <= size and threads:acceptsjob() do
+         local indices = perm:narrow(1, idx, math.min(batchSize, size - idx + 1))
+         threads:addjob(
+            function(indices, nCrops, cpuType)
+               local sz = indices:size(1)
+               local batch, imageSize
+               local target = torch.IntTensor(sz)
+               for i, idx in ipairs(indices:totable()) do
+                  local sample = _G.dataset:get(idx)
+                  local input = _G.preprocess(sample.input)
+                  if not batch then
+                     imageSize = input:size():totable()
+                     if nCrops > 1 then table.remove(imageSize, 1) end
+                     batch = torch[cpuType](sz, nCrops, table.unpack(imageSize))
+                  end
+                  batch[i]:copy(input)
+                  target[i] = sample.target
+               end
+               collectgarbage()
+               return {
+                  input = batch:view(sz * nCrops, table.unpack(imageSize)),
+                  target = target,
+               }
+            end,
+            function(_sample_)
+               sample = _sample_
+            end,
+            indices,
+            self.nCrops,
+            self.cpuType
+         )
+         idx = idx + batchSize
+      end
+   end
+
+   local n = 0
+   local function loop()
+      enqueue()
+      if not threads:hasjob() then
+         return nil
+      end
+      threads:dojob()
+      if threads:haserror() then
+         threads:synchronize()
+      end
+      enqueue()
+      n = n + 1
+      return n, sample
+   end
+
+   return loop
+end
+
+return M.DataLoader