update for pg 0.11.0 and a few minor changes

.gitignore

@@ -2,5 +2,8 @@
 configs
 test/fixtures/test*.sql
 pkg
+.redcar
 .rvmrc
 .bundle
+*.out
+rdoc

CHANGELOG

@@ -0,0 +1,19 @@
+0.2.2 Feb 13, 2012
+===
+
+* merge contributions from: James Coleman, Aaron Peckham, James Tippett
+* breaking change: note that "supress*" configuration parameters now use correct spelling. Existing config files you may have will need housekeeping.
+* new configuration options: suppress_sequence_update; suppress_indexes; use_timezones
+* adds a Gemfile to help setup development environment
+
+0.1.1 May 3, 2011
+===
+
+* updated for pg driver ~> 0.11.0
+* add a check for mysql cmd, e.g. mysql or mysql5
+* remove deprecated feature per gem -v 1.7.2
+
+0.1.0 September 18, 2010
+===
+
+* Initial packaging as a gem

Gemfile

@@ -1,6 +1,8 @@
 source :rubygems
 
-gem 'mysql'
-gem 'pg'
-gem 'test-unit'
-gem 'jeweler'
+group :development do
+  gem "bundler", "~> 1.0.21"
+  gem "jeweler", "~> 1.5.2"
+end
+
+gemspec

Gemfile.lock

@@ -1,21 +1,34 @@
+PATH
+  remote: .
+  specs:
+    mysql2psql (0.2.0)
+      mysql (= 2.8.1)
+      mysql2psql
+      pg (~> 0.11.0)
+
 GEM
   remote: http://rubygems.org/
   specs:
     git (1.2.5)
-    jeweler (1.6.0)
+    jeweler (1.5.2)
       bundler (~> 1.0.0)
       git (>= 1.2.5)
       rake
+    json (1.6.5)
     mysql (2.8.1)
     pg (0.11.0)
-    rake (0.8.7)
-    test-unit (2.3.0)
+    rake (0.9.2.2)
+    rdoc (3.12)
+      json (~> 1.4)
+    test-unit (2.4.7)
 
 PLATFORMS
   ruby
 
 DEPENDENCIES
-  jeweler
-  mysql
-  pg
-  test-unit
+  bundler (~> 1.0.21)
+  jeweler (~> 1.5.2)
+  mysql2psql!
+  rake (~> 0.9.2.2)
+  rdoc (~> 3.12)
+  test-unit (>= 2.1.1)

README.rdoc

@@ -11,9 +11,8 @@ to contact me, I'll help you.
 Mysql2psql is packaged as a gem. Install as usual (use sudo if required).
 
   $ gem install mysql2psql
-  NB: the gem hasn't been formally released on http://rubygems.org/ yet. For now you need to clone the git repo at http://github.com/tardate/mysql2postgres and 'rake install' 
 
-After installation, the "mysql2psql" command will be available. 
+After installation, the "mysql2psql" command will be available.
 When run, it will generate a default configuration file in the current directory if a config
 file is not already found. The default configuration filename is mysql2psql.yml.
 
@@ -42,83 +41,93 @@ After editing the configuration file and launching tool, you will see something
 
 
 == Database driver dependencies
-Mysql2psql uses the 'mysql' and 'pg' gems for database connectivity. 
+Mysql2psql uses the 'mysql' and 'pg' gems for database connectivity.
 Mysql2psql gem installation should automatically install these too.
 
 If you encounter any issues with db connectivity, verify that the 'mysql' and 'pg' gems are installed and working correctly first.
 The 'mysql' gem in particular may need a hint on where to find the mysql headers and libraries:
 
   $ [sudo] gem install mysql -- --with-mysql-dir=/usr/local/mysql
-
-NB: With Ruby 1.8.x, the gem install will usually complain about "No definition for ..." errors. These are non-fatal and just affect the documentation install. This doesn't happen with Ruby 1.9.2.
-
-With bundler you can do a simple: bundle install
 
+NB: With Ruby 1.8.x, the gem install will usually complain about "No definition for ..." errors. These are non-fatal and just affect the documentation install. This doesn't happen with Ruby 1.9.2.
+
+== Contributing to Mysql2psql
+
+* Check out the latest master to make sure the feature hasn't been implemented or the bug hasn't been fixed yet
+* Check out the issue tracker to make sure someone already hasn't requested it and/or contributed it
+* Fork the project
+* Start a feature/bugfix branch
+* Commit and push until you are happy with your contribution
+* Make sure to add tests for it. This is important so no-one unintentionally breaks it in a future version.
+* Please try not to mess with the Rakefile, version, or history. If you want to have your own version, or is otherwise necessary, that is fine, but please isolate to its own commit so the gem maintainers can cherry-pick around it.
+* Send a pull request
+
 == Getting the source
-Mysql2psql was first produced by Max Lapshin <[email protected]> who maintains the master 
-repository at http://github.com/maxlapshin/mysql2postgres
+Mysql2psql was first produced by Max Lapshin <[email protected]> who maintains the master
+repository at https://github.com/maxlapshin/mysql2postgres
 
-The gem packaged version you are currently looking at has yet to be merged into the master repo.
 The gem bundling and refactoring for testing was done by Paul Gallagher <[email protected]> and
-the repository is at http://github.com/tardate/mysql2postgres
+the repository is at https://github.com/tardate/mysql2postgres
 
+Gem packaging is done with Jeweler. Note that on windows this means you must run under mingw or other shell that supports git to do the gem bundling (but you can gem install the built gem in the normal Windows shell).
 
+== Setting up for development
+
+The project includes a Gemfile so with bundler you can install gem dependencies with:
+
+  gem install bundler # if not already installed
+  bundle install
+
+Note comments 'Database driver dependencies' regarding issues that you may encounter with the mysql gem.
+
 == Running tests
 If you fork/clone the project, you will want to run the test suite (and add to it if you are developing new features or fixing bugs).
- 
+
 A suite of tests are provided and are run using rake (rake -T for more info)
   rake test:units
   rake test:integration
-  rake test 
+  rake test
 
 Rake without parameters (or "rake test") will run both the unit and integration tests.
 
 Unit tests are small standalone tests of the mysql2psql codebase that have no external dependencies (like a database)
 
-Integration tests require suitable mysql and postgres databases to be setup in advance. 
+Integration tests require suitable mysql and postgres databases to be setup in advance.
 Running the integration tests *will* rewrite data and schema for the "mysql2psql_test" databases in mysql and postgres. Don't use this database for anything else!
 
-mysql on localhost:3306
-- database created called "mysql2psql_test"
-- user setup for "mysql2psql" with no password
-- e.g. 
+Setting up mysql on localhost:3306
+- create a database called "mysql2psql_test"
+- setup a user "mysql2psql" with no password
+
+e.g.
+
   mysqladmin -uroot -p create mysql2psql_test
   mysql -uroot -p -e "grant all on mysql2psql_test.* to 'mysql2psql'@'localhost';"
-  # verify connecction:
+  # verify connection:
   mysql -umysql2psql -e "select database(),'yes' as connected" mysql2psql_test
-
-postgres on localhost:5432
-- database created called "mysql2psql_test" 
-- role (user) access setup for "mysql2psql" with no password
-- e.g.
+
+
+Setting up PostgreSQL on localhost:5432
+- create a database called "mysql2psql_test"
+- setup a role (user) "mysql2psql" with no password
+
+e.g.
+
   psql postgres -c "create role mysql2psql with login"
   psql postgres -c "create database mysql2psql_test with owner mysql2psql encoding='UTF8'"
   # verify connection:
   psql mysql2psql_test -U mysql2psql -c "\c"
 
-== Notes, Limitations, Outstanding Issues..
 
-Todo:
-- more tests
-- release gem
-- a windows cmd shim
+== Notes, Limitations, Outstanding Issues..
 
-=== note from mgkimsal
-I'm still having trouble with bit(1)/boolean fields
-workaround I've found is to put output in file
-then in VIM on file, search/replace the true/false binary fields with t/f
-specifically
+* more test coverage, as always...
 
-(reversed on 3/23 - should be ^A gets f)
-:%s/^@/t/g
-:%s/^A/f/g
-keystrokes are ctrl-v ctrl-shift-@ to get the 'true' binary field
-keystrokes are ctrl-v ctrl-shift-A to get the 'false' binary field
 
 == Contributors
 Project founded by Max Lapshin <[email protected]>
 
-Other contributors (in git log order):
+Contributors (roughly git log order):
 - Anton Ageev <[email protected]>
 - Samuel Tribehou <[email protected]>
 - Marco Nenciarini <[email protected]>
@@ -130,8 +139,10 @@ Other contributors (in git log order):
 - Jacob Coby <[email protected]>
 - Neszt Tibor <[email protected]>
 - Miroslav Kratochvil <[email protected]>
-- Paul Gallagher <[email protected]>
-
-
-
-
+- {Paul Gallagher}[https://github.com/tardate] <[email protected]>
+- {James Coleman}[https://github.com/jcoleman] <[email protected]>
+- {Aaron Peckham}[https://github.com/apeckham]
+- {James Tippett}[https://github.com/jtippett]
+- {Tim Morgan}[https://github.com/seven1m]
+- {dakhota}[https://github.com/dakhota]
+- {Matthew Soldo}[https://github.com/mattsoldo]

Rakefile

@@ -13,7 +13,7 @@ begin
     gem.description = %Q{It can create postgresql dump from mysql database or directly load data from mysql to
     postgresql (at about 100 000 records per minute). Translates most data types and indexes.}
     gem.email = "[email protected]"
-    gem.homepage = "http://github.com/tardate/mysql2postgresql"
+    gem.homepage = "https://github.com/tardate/mysql2postgres"
     gem.authors = [
       "Max Lapshin <[email protected]>",
       "Anton Ageev <[email protected]>",
@@ -27,11 +27,19 @@ begin
       "Jacob Coby <[email protected]>",
       "Neszt Tibor <[email protected]>",
       "Miroslav Kratochvil <[email protected]>",
-      "Paul Gallagher <[email protected]>"
-      ]
+      "Paul Gallagher <[email protected]>",
+      "James Coleman <[email protected]>",
+      "Aaron Peckham",
+      "James Tippett",
+      "Tim Morgan",
+      "dakhota",
+      "Matthew Soldo"
+    ]
     gem.add_dependency "mysql", "= 2.8.1"
-    gem.add_dependency "pg", "= 0.9.0"
+    gem.add_dependency "pg", "~> 0.11.0"
     gem.add_development_dependency "test-unit", ">= 2.1.1"
+    gem.add_development_dependency "rake", "~> 0.9.2.2"
+    gem.add_development_dependency "rdoc", "~> 3.12"
     # gem is a Gem::Specification... see http://www.rubygems.org/read/chapter/20 for additional settings
   end
   Jeweler::GemcutterTasks.new
@@ -63,25 +71,25 @@ end
 begin
   require 'rcov/rcovtask'
   Rcov::RcovTask.new do |test|
-    test.libs << 'test'
+    test.libs << 'lib'
+    test.libs << 'test/lib'
     test.pattern = 'test/**/*test.rb'
     test.verbose = true
+    test.rcov_opts << "--exclude gems/*"
   end
 rescue LoadError
   task :rcov do
     abort "RCov is not available. In order to run rcov, you must: sudo gem install spicycode-rcov"
   end
 end
 
-
 task :default => :test
 
-require 'rake/rdoctask'
-Rake::RDocTask.new do |rdoc|
+require 'rdoc/task'
+RDoc::Task.new do |rdoc|
   version = Mysql2psql::Version::STRING
 
   rdoc.rdoc_dir = 'rdoc'
   rdoc.title = "mysql2psql #{version}"
-  rdoc.rdoc_files.include('README*')
-  rdoc.rdoc_files.include('lib/**/*.rb')
+  rdoc.rdoc_files.include("README*", "lib/**/*.rb")
 end

lib/mysql2psql/config.rb

@@ -1,13 +1,13 @@
 require 'mysql2psql/config_base'
 
 class Mysql2psql
-  
+
   class Config < ConfigBase
-    
+
     def initialize(configfilepath, generate_default_if_not_found = true)
       unless File.exists?(configfilepath)
         reset_configfile(configfilepath) if generate_default_if_not_found
-        if File.exists?(configfilepath) 
+        if File.exists?(configfilepath)
           raise Mysql2psql::ConfigurationFileInitialized.new("\n
 No configuration file found.
 A new file has been initialized at: #{configfilepath}
@@ -27,31 +27,42 @@ def reset_configfile(filepath)
       file.close
     end
 
-    def self.template(to_filename = nil, include_tables = [], exclude_tables = [], supress_data = false, supress_ddl = false, force_truncate = false)
+    # Returns template file text given +options+ hash which may include:
+    # :to_filename - default: nil
+    # :include_tables - default: []
+    # :exclude_tables - default: []
+    # :suppress_data - default: false
+    # :suppress_ddl - default: false
+    # :suppress_sequence_update - default: false
+    # :suppress_indexes - default: false
+    # :force_truncate - default: false
+    # :use_timezones - default: false
+    def self.template(options={})
       configtext = <<EOS
 mysql:
  hostname: localhost
  port: 3306
- socket: /tmp/mysql.sock
+ socket:
  username: mysql2psql
- password: 
+ password:
  database: mysql2psql_test
 
 destination:
  # if file is given, output goes to file, else postgres
- file: #{ to_filename ? to_filename : ''}
+ file: #{ options[:to_filename] || ''}
  postgres:
   hostname: localhost
   port: 5432
   username: mysql2psql
-  password: 
+  password:
   database: mysql2psql_test
 
 # if tables is given, only the listed tables will be converted.  leave empty to convert all tables.
 #tables:
 #- table1
 #- table2
 EOS
+      include_tables = options[:include_tables] || []
       if include_tables.length>0
         configtext += "\ntables:\n"
         include_tables.each do |t|
@@ -65,36 +76,67 @@ def self.template(to_filename = nil, include_tables = [], exclude_tables = [], s
 #- table4
 
 EOS
+      exclude_tables = options[:exclude_tables] || []
       if exclude_tables.length>0
         configtext += "\nexclude_tables:\n"
         exclude_tables.each do |t|
           configtext += "- #{t}\n"
         end
       end
-      if !supress_data.nil?
+
+      if !options[:suppress_data].nil?
+        configtext += <<EOS
+
+# if suppress_data is true, only the schema definition will be exported/migrated, and not the data
+suppress_data: #{options[:suppress_data]}
+EOS
+      end
+
+      if !options[:suppress_ddl].nil?
         configtext += <<EOS
 
-# if supress_data is true, only the schema definition will be exported/migrated, and not the data
-supress_data: #{supress_data}
+# if suppress_ddl is true, only the data will be exported/imported, and not the schema
+suppress_ddl: #{options[:suppress_ddl]}
 EOS
       end
-      if !supress_ddl.nil?
+
+      if !options[:suppress_sequence_update].nil?
         configtext += <<EOS
 
-# if supress_ddl is true, only the data will be exported/imported, and not the schema
-supress_ddl: #{supress_ddl}
+# if suppress_sequence_update is true, the sequences for serial (auto-incrementing) columns
+# will not be update to the current maximum value of that column in the database
+# if suppress_ddl is not set to true, then this option is implied to be false as well (unless overridden here)
+suppress_sequence_update: #{options[:suppress_sequence_update]}
 EOS
       end
-      if !force_truncate.nil?
+
+      if !options[:force_truncate].nil?
         configtext += <<EOS
 
 # if force_truncate is true, forces a table truncate before table loading
-force_truncate: #{force_truncate}
+force_truncate: #{options[:force_truncate]}
+EOS
+      end
+
+      if !options[:use_timezones].nil?
+        configtext += <<EOS
+        
+# if use_timezones is true, timestamp/time columns will be created in postgres as "with time zone"
+# rather than "without time zone"
+use_timezones: #{options[:use_timezones]}
+EOS
+      end
+
+      if !options[:suppress_indexes].nil?
+        configtext += <<EOS
+
+# if suppress_indexes is true, indexes will not be exported/migrated
+suppress_indexes: #{options[:suppress_indexes]}
 EOS
       end
       configtext
     end
-    
+
   end
 
 end

lib/mysql2psql/converter.rb

@@ -2,52 +2,59 @@ class Mysql2psql
 
   class Converter
     attr_reader :reader, :writer, :options
-    attr_reader :exclude_tables, :only_tables, :supress_data, :supress_ddl, :force_truncate
-  
+    attr_reader :exclude_tables, :only_tables, :suppress_data, :suppress_ddl, :supress_sequence_update, :force_truncate, :suppress_indexes
+
     def initialize(reader, writer, options)
       @reader = reader
       @writer = writer
       @options = options
       @exclude_tables = options.exclude_tables([])
       @only_tables = options.only_tables(nil)
-      @supress_data = options.supress_data(false)
-      @supress_ddl = options.supress_ddl(false)
+      @suppress_data = options.suppress_data(false)
+      @suppress_ddl = options.suppress_ddl(false)
+      @supress_sequence_update = options.supress_sequence_update(false)
+      @suppress_indexes = options.suppress_indexes(false)
       @force_truncate = options.force_truncate(false)
+      @use_timezones = options.use_timezones(false)
     end
-  
+
     def convert
       _time1 = Time.now
 
       tables = reader.tables.
         reject {|table| @exclude_tables.include?(table.name)}.
         select {|table| @only_tables ? @only_tables.include?(table.name) : true}
 
+      tables.each do |table|
+        writer.write_sequence_update(table, options)
+      end if !(@suppress_sequence_update && @suppress_ddl)
 
       tables.each do |table|
-        writer.write_table(table)
-      end unless @supress_ddl
+        writer.write_table(table, {:use_timezones => @use_timezones})
+      end unless @suppress_ddl
 
       _time2 = Time.now
       tables.each do |table|
-        writer.truncate(table) if force_truncate && supress_ddl
+        writer.truncate(table) if force_truncate && !suppress_ddl
         writer.write_contents(table, reader)
-      end unless @supress_data
- 
+      end unless @suppress_data
+
       _time3 = Time.now
       tables.each do |table|
-        writer.write_indexes(table)
-      end unless @supress_ddl
+        writer.write_indexes(table) unless @suppress_indexes
+      end unless @suppress_ddl
       tables.each do |table|
         writer.write_constraints(table)
-      end unless @supress_ddl
- 
- 
+      end unless @suppress_ddl
+
+
       writer.close
       _time4 = Time.now
       puts "Table creation #{((_time2 - _time1) / 60).round} min, loading #{((_time3 - _time2) / 60).round} min, indexing #{((_time4 - _time3) / 60).round} min, total #{((_time4 - _time1) / 60).round} min"
       return 0
     rescue => e
       $stderr.puts "Mysql2psql: conversion failed: #{e.to_s}"
+      $stderr.puts e.backtrace
       return -1
     end
   end

lib/mysql2psql/mysql_reader.rb

@@ -28,12 +28,16 @@ def columns
 
       def convert_type(type)
         case type
-        when /int.* unsigned/
+        when /^int.* unsigned/
           "bigint"
         when /bigint/
           "bigint"
         when "bit(1)"
           "boolean"
+        when /smallint.* unsigned/
+          "integer"
+        when /smallint/
+          "smallint"
         when "tinyint(1)"
           "boolean"
         when /tinyint/
@@ -44,13 +48,15 @@ def convert_type(type)
           "varchar"
         when /char/
           "char"
-        when /(float|decimal)/
+        when /decimal/
           "decimal"
-        when /double/
-           "double precision"
+        when /float/
+          "float"
+        when /real|double/
+          "double precision"
         else
           type
-        end 
+        end
       end
 
       def load_columns
@@ -60,6 +66,7 @@ def load_columns
         fields = []
         @reader.mysql.query("EXPLAIN `#{name}`") do |res|
           while field = res.fetch_row do
+            length = -1
             length = field[1][/\((\d+)\)/, 1] if field[1] =~ /\((\d+)\)/
             length = field[1][/\((\d+),(\d+)\)/, 1] if field[1] =~ /\((\d+),(\d+)\)/
             desc = {
@@ -88,7 +95,7 @@ def load_columns
 
       def indexes
         load_indexes unless @indexes
-        @indexes 
+        @indexes
       end
 
       def foreign_keys
@@ -147,7 +154,7 @@ def count_rows
       end
 
       def has_id?
-        !!columns.find {|col| col[:name] == "id"} 
+        !!columns.find {|col| col[:name] == "id"}
       end
 
       def count_for_pager
@@ -175,9 +182,9 @@ def reconnect
     end
 
     def initialize(options)
-      @host, @user, @passwd, @db, @port, @sock, @flag = 
-        options.mysqlhostname('localhost'), options.mysqlusername, 
-        options.mysqlpassword, options.mysqldatabase, 
+      @host, @user, @passwd, @db, @port, @sock, @flag =
+        options.mysqlhostname('localhost'), options.mysqlusername,
+        options.mysqlpassword, options.mysqldatabase,
         options.mysqlport, options.mysqlsocket
       @port = nil if @port == ""  # for things like Amazon's RDS you don't have a port and connect fails with "" for a value
       @sock = nil if @sock == ""
@@ -186,9 +193,22 @@ def initialize(options)
     end
 
     attr_reader :mysql
-
+
+    def views
+      unless defined? @views
+        @mysql.query("SELECT t.TABLE_NAME FROM INFORMATION_SCHEMA.TABLES t WHERE t.TABLE_SCHEMA = '#{@db}' AND t.TABLE_TYPE = 'VIEW';") do |res|
+          @views = []
+          res.each { |row| @views << row[0] }
+        end
+      end
+
+      @views
+    end
+
     def tables
-      @tables ||= @mysql.list_tables.map {|table| Table.new(self, table)}
+      @tables ||= (@mysql.list_tables - views).map do |table|
+        Table.new(self, table)
+      end
     end
 
     def paginated_read(table, page_size)
@@ -207,4 +227,4 @@ def paginated_read(table, page_size)
     end
   end
 
-end
+end

lib/mysql2psql/postgres_db_writer.rb

@@ -8,8 +8,8 @@ class PostgresDbWriter < PostgresWriter
   attr_reader :conn, :hostname, :login, :password, :database, :schema, :port
 
   def initialize(options)
-    @hostname, @login, @password, @database, @port =  
-      options.pghostname('localhost'), options.pgusername, 
+    @hostname, @login, @password, @database, @port =
+      options.pghostname('localhost'), options.pgusername,
       options.pgpassword, options.pgdatabase, options.pgport(5432).to_s
     @database, @schema = database.split(":")
     open
@@ -21,53 +21,61 @@ def open
     @conn.exec("SET client_encoding = 'UTF8'")
     @conn.exec("SET standard_conforming_strings = off") if @conn.server_version >= 80200
     @conn.exec("SET check_function_bodies = false")
-    @conn.exec("SET client_min_messages = warning")    
+    @conn.exec("SET client_min_messages = warning")
   end
 
   def close
     @conn.close
   end
-
+  
   def exists?(relname)
     rc = @conn.exec("SELECT COUNT(*) FROM pg_class WHERE relname = '#{relname}'")
     (!rc.nil?) && (rc.to_a.length==1) && (rc.first.count.to_i==1)
   end
 
-  def write_table(table)
+  def write_sequence_update(table, options)
+    serial_key_column = table.columns.detect do |column|
+      column[:auto_increment]
+    end
+
+    if serial_key_column
+      serial_key = serial_key_column[:name]
+      max_value = serial_key_column[:maxval].to_i < 1 ? 1 : serial_key_column[:maxval] + 1
+      serial_key_seq = "#{table.name}_#{serial_key}_seq"
+
+      if !options.supress_ddl
+        if @conn.server_version < 80200
+          @conn.exec("DROP SEQUENCE #{serial_key_seq} CASCADE") if exists?(serial_key_seq)
+        else
+          @conn.exec("DROP SEQUENCE IF EXISTS #{serial_key_seq} CASCADE")
+        end
+        @conn.exec <<-EOF
+          CREATE SEQUENCE #{serial_key_seq}
+          INCREMENT BY 1
+          NO MAXVALUE
+          NO MINVALUE
+          CACHE 1
+        EOF
+      end
+
+      if !options.supress_sequence_update
+        puts "Updated sequence #{serial_key_seq} to current value of #{max_value}"
+        @conn.exec sqlfor_set_serial_sequence(table, serial_key_seq, max_value)
+      end
+    end
+  end
+
+  def write_table(table, options)
     puts "Creating table #{table.name}..."
     primary_keys = []
-    serial_key = nil
-    maxval = nil
 
     columns = table.columns.map do |column|
-      if column[:auto_increment]
-        serial_key = column[:name]
-        maxval = column[:maxval].to_i < 1 ? 1 : column[:maxval] + 1
-      end
       if column[:primary_key]
         primary_keys << column[:name]
       end
-      "  " + column_description(column)
+      "  " + column_description(column, options)
     end.join(",\n")
 
-    if serial_key
-      if @conn.server_version < 80200
-        serial_key_seq = "#{table.name}_#{serial_key}_seq"
-        @conn.exec("DROP SEQUENCE #{serial_key_seq} CASCADE") if exists?(serial_key_seq)
-      else
-        @conn.exec("DROP SEQUENCE IF EXISTS #{table.name}_#{serial_key}_seq CASCADE")
-      end
-      @conn.exec <<-EOF
-        CREATE SEQUENCE #{table.name}_#{serial_key}_seq
-        INCREMENT BY 1
-        NO MAXVALUE
-        NO MINVALUE
-        CACHE 1
-      EOF
-
-      @conn.exec "SELECT pg_catalog.setval('#{table.name}_#{serial_key}_seq', #{maxval}, true)"
-    end
-
     if @conn.server_version < 80200
       @conn.exec "DROP TABLE #{PGconn.quote_ident(table.name)} CASCADE;" if exists?(table.name)
     else
@@ -83,42 +91,48 @@ def write_table(table)
     puts "Created table #{table.name}"
 
   end
-  
+
   def write_indexes(table)
     puts "Indexing table #{table.name}..."
     if primary_index = table.indexes.find {|index| index[:primary]}
-      @conn.exec("ALTER TABLE #{PGconn.quote_ident(table.name)} ADD CONSTRAINT \"#{table.name}_pkey\" PRIMARY KEY(#{primary_index[:columns].map {|col| PGconn.quote_ident(col)}.join(", ")})")
+      index_sql = "ALTER TABLE #{PGconn.quote_ident(table.name)} ADD CONSTRAINT \"#{table.name}_pkey\" PRIMARY KEY(#{primary_index[:columns].map {|col| PGconn.quote_ident(col)}.join(", ")})"
+      @conn.exec(index_sql)
     end
-    
+
     table.indexes.each do |index|
       next if index[:primary]
       unique = index[:unique] ? "UNIQUE " : nil
 
-      #MySQL allows an index name which could be equal to a table name, Postgres doesn't
+      # MySQL allows an index name which could be equal to a table name, Postgres doesn't
       indexname = index[:name]
+      indexname_quoted = ''
+
       if indexname.eql?(table.name)
-        indexnamenew = "#{indexname}_index"
-        puts "WARNING: index \"#{indexname}\" equals table name. This is not allowed by postgres and will be renamed to \"#{indexnamenew}\""
-        indexname = indexnamenew
+        indexname = (@conn.server_version < 90000) ? "#{indexname}_index" : nil
+        puts "WARNING: index \"#{index[:name]}\" equals table name. This is not allowed in PostgreSQL and will be renamed."
       end
-
-      if @conn.server_version < 80200
-        @conn.exec("DROP INDEX #{PGconn.quote_ident(indexname)} CASCADE;") if exists?(indexname)
-      else
-        @conn.exec("DROP INDEX IF EXISTS #{PGconn.quote_ident(indexname)} CASCADE;")
+
+      if indexname
+        indexname_quoted = PGconn.quote_ident(indexname)
+        if @conn.server_version < 80200
+          @conn.exec("DROP INDEX #{PGconn.quote_ident(indexname)} CASCADE;") if exists?(indexname)
+        else
+          @conn.exec("DROP INDEX IF EXISTS #{PGconn.quote_ident(indexname)} CASCADE;")
+        end
       end
-      @conn.exec("CREATE #{unique}INDEX #{PGconn.quote_ident(indexname)} ON #{PGconn.quote_ident(table.name)} (#{index[:columns].map {|col| PGconn.quote_ident(col)}.join(", ")});")
+
+      index_sql = "CREATE #{unique}INDEX #{indexname_quoted} ON #{PGconn.quote_ident(table.name)} (#{index[:columns].map {|col| PGconn.quote_ident(col)}.join(", ")});"
+      @conn.exec(index_sql)
     end
-
-
+
     #@conn.exec("VACUUM FULL ANALYZE #{PGconn.quote_ident(table.name)}")
     puts "Indexed table #{table.name}"
   rescue Exception => e
     puts "Couldn't create indexes on #{table} (#{table.indexes.inspect})"
     puts e
     puts e.backtrace[0,3].join("\n")
   end
-  
+
   def write_constraints(table)
     table.foreign_keys.each do |key|
       key_sql = "ALTER TABLE #{PGconn.quote_ident(table.name)} ADD FOREIGN KEY (#{key[:column].map{|c|PGconn.quote_ident(c)}.join(', ')}) REFERENCES #{PGconn.quote_ident(key[:ref_table])}(#{key[:ref_column].map{|c|PGconn.quote_ident(c)}.join(', ')}) ON UPDATE #{key[:on_update]} ON DELETE #{key[:on_delete]}"
@@ -149,7 +163,6 @@ def write_contents(table, reader)
     puts "Loading #{table.name}..."
     STDOUT.flush
     _counter = reader.paginated_read(table, 1000) do |row, counter|
-      line = []
       process_row(table, row)
       @conn.put_copy_data(row.join("\t") + "\n")
 
@@ -164,16 +177,25 @@ def write_contents(table, reader)
 
       if counter % 5000 == 0
         @conn.put_copy_end
+        res = @conn.get_result
+        if res.cmdtuples != 5000
+          puts "\nWARNING: #{table.name} expected 5000 tuple inserts got #{res.cmdtuples} at row #{counter}\n"
+        end
         @conn.exec(copy_line)
       end
 
     end
-    _time2 = Time.now
-    puts "\n#{_counter} rows loaded in #{((_time2 - _time1) / 60).round}min #{((_time2 - _time1) % 60).round}s"
-#    @conn.putline(".\n")
     @conn.put_copy_end
+    if _counter && (_counter % 5000) > 0
+      res = @conn.get_result
+      if res.cmdtuples != (_counter % 5000)
+        puts "\nWARNING: table #{table.name} expected #{_counter % 5000} tuple inserts got #{res.cmdtuples}\n"
+      end
+    end
+    _time2 = Time.now
+    puts "\n#{table.name} #{_counter} rows loaded in #{((_time2 - _time1) / 60).round}min #{((_time2 - _time1) % 60).round}s"
   end
-  
+
 end
 
-end
+end

lib/mysql2psql/postgres_file_writer.rb

@@ -33,47 +33,58 @@ def truncate(table)
 EOF
     if serial_key
     @f << <<-EOF
-SELECT pg_catalog.setval(pg_get_serial_sequence('#{table.name}', '#{serial_key}'), #{maxval}, true);
+#{sqlfor_reset_serial_sequence(table,serial_key,maxval)}
 EOF
     end
   end
 
-  def write_table(table)
-    primary_keys = []
-    serial_key = nil
-    maxval = nil
-
-    columns = table.columns.map do |column|
-      if column[:auto_increment]
-        serial_key = column[:name]
-        maxval = column[:maxval].to_i < 1 ? 1 : column[:maxval] + 1
-      end
-      if column[:primary_key]
-        primary_keys << column[:name]
-      end
-      "  " + column_description(column)
-    end.join(",\n")
+  def write_sequence_update(table, options)
+    serial_key_column = table.columns.detect do |column|
+      column[:auto_increment]
+    end
 
-    if serial_key
+    if serial_key_column
+      serial_key = serial_key_column[:name]
+      serial_key_seq = "#{table.name}_#{serial_key}_seq"
+      max_value = serial_key_column[:maxval].to_i < 1 ? 1 : serial_key_column[:maxval] + 1
 
       @f << <<-EOF
 --
--- Name: #{table.name}_#{serial_key}_seq; Type: SEQUENCE; Schema: public
+-- Name: #{serial_key_seq}; Type: SEQUENCE; Schema: public
 --
+EOF
+
+      if !options.supress_ddl
+        @f << <<-EOF
+DROP SEQUENCE IF EXISTS #{serial_key_seq} CASCADE;
  
-DROP SEQUENCE IF EXISTS #{table.name}_#{serial_key}_seq CASCADE;
- 
-CREATE SEQUENCE #{table.name}_#{serial_key}_seq
+CREATE SEQUENCE #{serial_key_seq}
     INCREMENT BY 1
     NO MAXVALUE
     NO MINVALUE
     CACHE 1;
-    
-    
-SELECT pg_catalog.setval('#{table.name}_#{serial_key}_seq', #{maxval}, true);
- 
-      EOF
+EOF
+      end
+
+      if !options.supress_sequence_update
+        @f << <<-EOF
+#{sqlfor_set_serial_sequence(table, serial_key_seq, max_value)}
+EOF
+      end
     end
+  end
+
+  def write_table(table, options)
+    primary_keys = []
+    serial_key = nil
+    maxval = nil
+
+    columns = table.columns.map do |column|
+      if column[:primary_key]
+        primary_keys << column[:name]
+      end
+      "  " + column_description(column, options)
+    end.join(",\n")
 
     @f << <<-EOF
 -- Table: #{table.name}
@@ -126,7 +137,6 @@ def write_contents(table, reader)
 EOF
 
     reader.paginated_read(table, 1000) do |row, counter|
-      line = []
       process_row(table, row)
       @f << row.join("\t") + "\n"
     end

lib/mysql2psql/postgres_writer.rb

@@ -5,139 +5,164 @@
 class Mysql2psql
 
   class PostgresWriter < Writer
-    def column_description(column)
-      "#{PGconn.quote_ident(column[:name])} #{column_type_info(column)}"
+    def column_description(column, options)
+      "#{PGconn.quote_ident(column[:name])} #{column_type_info(column, options)}"
     end
-
-    def column_type(column)
-      column_type_info(column).split(" ").first
-    end
-
-    def column_type_info(column)
+
+    def column_type(column, options={})
       if column[:auto_increment]
-        return "integer DEFAULT nextval('#{column[:table_name]}_#{column[:name]}_seq'::regclass) NOT NULL"
+        'integer'
+      else
+        case column[:type]
+        when 'char'
+          "character(#{column[:length]})"
+        when 'varchar'
+          "character varying(#{column[:length]})"
+        when /tinyint|smallint/
+          'smallint'
+        when 'real', /float/, 'double precision'
+          'double precision'
+        when 'decimal'
+          # TODO: seven1m thinks "real" instead?
+          "numeric(#{column[:length] || 10}, #{column[:decimals] || 5})"
+        when 'datetime', 'timestamp'
+          "timestamp with#{options[:use_timezones] ? '' : 'out'} time zone"
+        when 'time'
+          "time with#{options[:use_timezones] ? '' : 'out'} time zone"
+        when 'tinyblob', 'mediumblob', 'longblob', 'blob', 'varbinary'
+          'bytea'
+        when 'tinytext', 'mediumtext', 'longtext', 'text'
+          'text'
+        when /^enum/
+          enum = column[:type].gsub(/enum|\(|\)/, '')
+          max_enum_size = enum.split(',').map{ |check| check.size() -2}.sort[-1]
+          "character varying(#{max_enum_size}) check( #{column[:name]} in (#{enum}))"
+        when 'integer', 'bigint', 'boolean', 'date'
+          column[:type]
+        else
+          puts "Unknown #{column.inspect}"
+          ''
+        end
       end
+    end
 
-      default = column[:default] ? " DEFAULT #{column[:default] == nil ? 'NULL' : "'"+PGconn.escape(column[:default])+"'"}" : nil
-      null = column[:null] ? "" : " NOT NULL"
-      type = 
-      case column[:type]
-
-      # String types
-      when "char"
-        default = default + "::char" if default
-        "character(#{column[:length]})"
-      when "varchar"
-        default = default + "::character varying" if default
-  #      puts "VARCHAR: #{column.inspect}"
-        "character varying(#{column[:length]})"
-
-      # Integer and numeric types
-      when "integer"
-        default = " DEFAULT #{column[:default].nil? ? 'NULL' : column[:default].to_i}" if default
-        "integer"
-      when "bigint"
-        default = " DEFAULT #{column[:default].nil? ? 'NULL' : column[:default].to_i}" if default
-        "bigint"
-      when "tinyint"
-        default = " DEFAULT #{column[:default].nil? ? 'NULL' : column[:default].to_i}" if default
-        "smallint"
-
-      when "boolean"
-        default = " DEFAULT #{column[:default].to_i == 1 ? 'true' : 'false'}" if default
-        "boolean"
-      when "float"
-        default = " DEFAULT #{column[:default].nil? ? 'NULL' : column[:default].to_f}" if default
-        "real"
-      when "float unsigned"
-        default = " DEFAULT #{column[:default].nil? ? 'NULL' : column[:default].to_f}" if default
-        "real"
-      when "decimal"
-        default = " DEFAULT #{column[:default].nil? ? 'NULL' : column[:default]}" if default
-        "numeric(#{column[:length] || 10}, #{column[:decimals] || 0})"
-
-      when "double precision"
-        default = " DEFAULT #{column[:default].nil? ? 'NULL' : column[:default]}" if default
-        "double precision"
-
-      # Mysql datetime fields
-      when "datetime"
-        default = nil
-        "timestamp without time zone"
-      when "date"
-        default = nil
-        "date"
-      when "timestamp"
-        default = " DEFAULT CURRENT_TIMESTAMP" if column[:default] == "CURRENT_TIMESTAMP"
-        default = " DEFAULT '1970-01-01 00:00'" if column[:default] == "0000-00-00 00:00"
-        default = " DEFAULT '1970-01-01 00:00:00'" if column[:default] == "0000-00-00 00:00:00"
-        "timestamp without time zone"
-      when "time"
-        default = " DEFAULT NOW()" if default
-        "time without time zone"
+    def column_default(column)
+      if column[:auto_increment]
+        "nextval('#{column[:table_name]}_#{column[:name]}_seq'::regclass)"
+      elsif column[:default]
+        case column[:type]
+        when 'char'
+          "'#{PGconn.escape(column[:default])}'::char"
+        when 'varchar', /^enum/
+          "'#{PGconn.escape(column[:default])}'::character varying"
+        when 'integer', 'bigint', /tinyint|smallint/
+          column[:default].to_i
+        when 'real', /float/
+          column[:default].to_f
+        when 'decimal', 'double precision'
+          column[:default]
+        when 'boolean'
+          case column[:default]
+          when nil
+            'NULL'
+          when 0, '0', "b'0'"
+            'false'
+          else
+            # Case for 1, '1', "b'1'" (for BIT(1) the data type), or anything non-nil and non-zero (for the TINYINT(1) type)
+            'true'
+          end
+        when 'timestamp', 'datetime', 'date'
+          case column[:default]
+          when 'CURRENT_TIMESTAMP'
+            'CURRENT_TIMESTAMP'
+          when '0000-00-00'
+            "'1970-01-01'"
+          when '0000-00-00 00:00'
+            "'1970-01-01 00:00'"
+          when '0000-00-00 00:00:00'
+            "'1970-01-01 00:00:00'"
+          else
+            "'#{PGconn.escape(column[:default])}'"
+          end
+        when 'time'
+          "'#{PGconn.escape(column[:default])}'"
+        else
+          # TODO: column[:default] will never be nil here.
+          #       Perhaps we should also issue a warning if this case is encountered.
+          "#{column[:default] == nil ? 'NULL' : "'"+PGconn.escape(column[:default])+"'"}"
+        end
+      end
+    end
 
-      when "tinyblob"
-        "bytea"
-      when "mediumblob"
-        "bytea"
-      when "longblob"
-        "bytea"
-      when "blob"
-        "bytea"
-      when "varbinary"
-        "bytea"
-      when "tinytext"
-        "text"
-      when "mediumtext"
-        "text"
-      when "longtext"
-        "text"
-      when "text"
-        "text"
-      when /^enum/
-        default = default + "::character varying" if default
-        enum = column[:type].gsub(/enum|\(|\)/, '')
-        max_enum_size = enum.split(',').map{ |check| check.size() -2}.sort[-1]
-        "character varying(#{max_enum_size}) check( #{column[:name]} in (#{enum}))"
+    def column_type_info(column, options)
+      type = column_type(column, options)
+      if type
+        not_null = !column[:null] || column[:auto_increment] ? ' NOT NULL' : ''
+        default = column[:default] || column[:auto_increment] ? " DEFAULT #{column_default(column)}" : ''
+        "#{type}#{default}#{not_null}"
       else
-        puts "Unknown #{column.inspect}"
-        column[:type].inspect
-        return ""
+        ''
       end
-      "#{type}#{default}#{null}"
     end
-
-    def process_row(table, row)
-    	table.columns.each_with_index do |column, index|
 
-          if column[:type] == "time"
+    def process_row(table, row)
+      table.columns.each_with_index do |column, index|
+        if column[:type] == 'time'
+          begin
             row[index] = "%02d:%02d:%02d" % [row[index].hour, row[index].minute, row[index].second]
+          rescue
+            # Don't fail on nil date/time.
           end
-
-          if row[index].is_a?(Mysql::Time)
-            row[index] = row[index].to_s.gsub('0000-00-00 00:00', '1970-01-01 00:00')
-            row[index] = row[index].to_s.gsub('0000-00-00 00:00:00', '1970-01-01 00:00:00')
-          end
-
-          if column_type(column) == "boolean"
-            row[index] = row[index] == 1 ? 't' : row[index] == 0 ? 'f' : row[index]
-          end
-
-          if row[index].is_a?(String)
-            if column_type(column) == "bytea"
-              row[index] = PGconn.escape_bytea(row[index])
+        elsif row[index].is_a?(Mysql::Time)
+          row[index] = row[index].to_s.gsub('0000-00-00 00:00', '1970-01-01 00:00')
+          row[index] = row[index].to_s.gsub('0000-00-00 00:00:00', '1970-01-01 00:00:00')
+        elsif column[:type] == 'boolean'
+          row[index] = (
+            case row[index]
+            when nil
+              '\N' # See note below about null values.
+            when 0, "\0"
+              'f'
+            else
+              # Case for 1, "\1" (for the BIT(1) data type), or anything non-nil and non-zero (to handle the TINYINT(1) type)
+              't'
+            end
+          )
+        elsif row[index].is_a?(String)
+          if column_type(column) == "bytea"
+            row[index] = PGconn.escape_bytea(row[index])
+          else
+            if row[index] == '\N' || row[index] == '\.'
+              row[index] = '\\' + row[index] # Escape our two PostgreSQL-text-mode-special strings.
             else
-              row[index] = row[index].gsub(/\\/, '\\\\\\').gsub(/\n/,'\n').gsub(/\t/,'\t').gsub(/\r/,'\r').gsub(/\0/, '')
+              # Awesome side-effect producing conditional. Don't do this at home.
+              unless row[index].gsub!(/\0/, '').nil?
+                puts "Removed null bytes from string since PostgreSQL TEXT types don't allow the storage of null bytes."
+              end
+
+              row[index] = row[index].dump
+              row[index] = row[index].slice(1, row[index].size-2)
             end
           end
-
-          row[index] = '\N' if !row[index]
+        elsif row[index].nil?
+          # Note: '\N' not "\N" is correct here:
+          #       The string containing the literal backslash followed by 'N'
+          #       represents database NULL value in PostgreSQL's text mode.
+          row[index] = '\N'
         end
+      end
     end
-  
+
     def truncate(table)
     end
-
+
+    def sqlfor_set_serial_sequence(table, serial_key_seq, max_value)
+      "SELECT pg_catalog.setval('#{serial_key_seq}', #{max_value}, true);"
+    end
+    def sqlfor_reset_serial_sequence(table, serial_key, max_value)
+      "SELECT pg_catalog.setval(pg_get_serial_sequence('#{table.name}', '#{serial_key}'), #{max_value}, true);"
+    end
+
   end
 
-end
+end

lib/mysql2psql/version.rb

@@ -1,7 +1,7 @@
 class Mysql2psql
   module Version
     MAJOR = 0
-    MINOR = 1
+    MINOR = 2
     PATCH = 0
 
     STRING = [MAJOR, MINOR, PATCH].compact.join('.')

mysql2psql.gemspec

@@ -1,15 +1,15 @@
 # Generated by jeweler
 # DO NOT EDIT THIS FILE DIRECTLY
-# Instead, edit Jeweler::Tasks in Rakefile, and run the gemspec command
+# Instead, edit Jeweler::Tasks in Rakefile, and run 'rake gemspec'
 # -*- encoding: utf-8 -*-
 
 Gem::Specification.new do |s|
   s.name = %q{mysql2psql}
-  s.version = "0.1.0"
+  s.version = "0.2.0"
 
   s.required_rubygems_version = Gem::Requirement.new(">= 0") if s.respond_to? :required_rubygems_version=
-  s.authors = ["Max Lapshin <max@maxidoors.ru>", "Anton Ageev <anton@ageev.name>", "Samuel Tribehou <cracoucax@gmail.com>", "Marco Nenciarini <marco.nenciarini@devise.it>", "James Nobis <jnobis@jnobis.controldocs.com>", "quel <github@quelrod.net>", "Holger Amann <keeney@fehu.org>", "Maxim Dobriakov <closer.main@gmail.com>", "Michael Kimsal <mgkimsal@gmail.com>", "Jacob Coby <jcoby@portallabs.com>", "Neszt Tibor <neszt@tvnetwork.hu>", "Miroslav Kratochvil <exa.exa@gmail.com>", "Paul Gallagher <gallagher.paul@gmail.com>"]
-  s.date = %q{2010-09-19}
+  s.authors = ["Max Lapshin <max@maxidoors.ru>", "Anton Ageev <anton@ageev.name>", "Samuel Tribehou <cracoucax@gmail.com>", "Marco Nenciarini <marco.nenciarini@devise.it>", "James Nobis <jnobis@jnobis.controldocs.com>", "quel <github@quelrod.net>", "Holger Amann <keeney@fehu.org>", "Maxim Dobriakov <closer.main@gmail.com>", "Michael Kimsal <mgkimsal@gmail.com>", "Jacob Coby <jcoby@portallabs.com>", "Neszt Tibor <neszt@tvnetwork.hu>", "Miroslav Kratochvil <exa.exa@gmail.com>", "Paul Gallagher <gallagher.paul@gmail.com>", "James Coleman <jtc331@gmail.com>", "Aaron Peckham", "James Tippett", "Tim Morgan", "dakhota", "Matthew Soldo"]
+  s.date = %q{2012-02-13}
   s.default_executable = %q{mysql2psql}
   s.description = %q{It can create postgresql dump from mysql database or directly load data from mysql to
     postgresql (at about 100 000 records per minute). Translates most data types and indexes.}
@@ -19,73 +19,145 @@ Gem::Specification.new do |s|
     "README.rdoc"
   ]
   s.files = [
-    ".gitignore",
-     "MIT-LICENSE",
-     "README.rdoc",
-     "Rakefile",
-     "bin/mysql2psql",
-     "lib/mysql2psql.rb",
-     "lib/mysql2psql/config.rb",
-     "lib/mysql2psql/config_base.rb",
-     "lib/mysql2psql/converter.rb",
-     "lib/mysql2psql/errors.rb",
-     "lib/mysql2psql/mysql_reader.rb",
-     "lib/mysql2psql/postgres_db_writer.rb",
-     "lib/mysql2psql/postgres_file_writer.rb",
-     "lib/mysql2psql/postgres_writer.rb",
-     "lib/mysql2psql/version.rb",
-     "lib/mysql2psql/writer.rb",
-     "mysql2psql.gemspec",
-     "test/fixtures/config_all_options.yml",
-     "test/fixtures/seed_integration_tests.sql",
-     "test/integration/convert_to_db_test.rb",
-     "test/integration/convert_to_file_test.rb",
-     "test/integration/converter_test.rb",
-     "test/integration/mysql_reader_base_test.rb",
-     "test/integration/mysql_reader_test.rb",
-     "test/integration/postgres_db_writer_base_test.rb",
-     "test/lib/ext_test_unit.rb",
-     "test/lib/test_helper.rb",
-     "test/units/config_base_test.rb",
-     "test/units/config_test.rb",
-     "test/units/postgres_file_writer_test.rb"
+    "CHANGELOG",
+    "Gemfile",
+    "Gemfile.lock",
+    "MIT-LICENSE",
+    "README.rdoc",
+    "Rakefile",
+    "bin/mysql2psql",
+    "lib/mysql2psql.rb",
+    "lib/mysql2psql/config.rb",
+    "lib/mysql2psql/config_base.rb",
+    "lib/mysql2psql/converter.rb",
+    "lib/mysql2psql/errors.rb",
+    "lib/mysql2psql/mysql_reader.rb",
+    "lib/mysql2psql/postgres_db_writer.rb",
+    "lib/mysql2psql/postgres_file_writer.rb",
+    "lib/mysql2psql/postgres_writer.rb",
+    "lib/mysql2psql/version.rb",
+    "lib/mysql2psql/writer.rb",
+    "mysql2psql.gemspec",
+    "test/fixtures/config_all_options.yml",
+    "test/fixtures/seed_integration_tests.sql",
+    "test/integration/convert_to_db_test.rb",
+    "test/integration/convert_to_file_test.rb",
+    "test/integration/converter_test.rb",
+    "test/integration/mysql_reader_base_test.rb",
+    "test/integration/mysql_reader_test.rb",
+    "test/integration/postgres_db_writer_base_test.rb",
+    "test/lib/ext_test_unit.rb",
+    "test/lib/test_helper.rb",
+    "test/units/config_base_test.rb",
+    "test/units/config_test.rb",
+    "test/units/postgres_file_writer_test.rb"
   ]
-  s.homepage = %q{http://github.com/tardate/mysql2postgresql}
-  s.rdoc_options = ["--charset=UTF-8"]
+  s.homepage = %q{https://github.com/tardate/mysql2postgres}
   s.require_paths = ["lib"]
-  s.rubygems_version = %q{1.3.7}
+  s.rubygems_version = %q{1.6.2}
   s.summary = %q{Tool for converting mysql database to postgresql}
   s.test_files = [
     "test/integration/convert_to_db_test.rb",
-     "test/integration/convert_to_file_test.rb",
-     "test/integration/converter_test.rb",
-     "test/integration/mysql_reader_base_test.rb",
-     "test/integration/mysql_reader_test.rb",
-     "test/integration/postgres_db_writer_base_test.rb",
-     "test/lib/ext_test_unit.rb",
-     "test/lib/test_helper.rb",
-     "test/units/config_base_test.rb",
-     "test/units/config_test.rb",
-     "test/units/postgres_file_writer_test.rb"
+    "test/integration/convert_to_file_test.rb",
+    "test/integration/converter_test.rb",
+    "test/integration/mysql_reader_base_test.rb",
+    "test/integration/mysql_reader_test.rb",
+    "test/integration/postgres_db_writer_base_test.rb",
+    "test/lib/ext_test_unit.rb",
+    "test/lib/test_helper.rb",
+    "test/units/config_base_test.rb",
+    "test/units/config_test.rb",
+    "test/units/postgres_file_writer_test.rb"
   ]
 
   if s.respond_to? :specification_version then
-    current_version = Gem::Specification::CURRENT_SPECIFICATION_VERSION
     s.specification_version = 3
 
     if Gem::Version.new(Gem::VERSION) >= Gem::Version.new('1.2.0') then
+      s.add_runtime_dependency(%q<mysql2psql>, [">= 0"])
+      s.add_development_dependency(%q<bundler>, ["~> 1.0.21"])
+      s.add_development_dependency(%q<jeweler>, ["~> 1.5.2"])
+      s.add_development_dependency(%q<bundler>, ["~> 1.0.21"])
+      s.add_development_dependency(%q<jeweler>, ["~> 1.5.2"])
+      s.add_development_dependency(%q<bundler>, ["~> 1.0.21"])
+      s.add_development_dependency(%q<jeweler>, ["~> 1.5.2"])
+      s.add_development_dependency(%q<test-unit>, [">= 2.1.1"])
+      s.add_development_dependency(%q<rake>, ["~> 0.9.2.2"])
+      s.add_development_dependency(%q<rdoc>, ["~> 3.12"])
+      s.add_development_dependency(%q<test-unit>, [">= 2.1.1"])
+      s.add_development_dependency(%q<rake>, ["~> 0.9.2.2"])
+      s.add_development_dependency(%q<rdoc>, ["~> 3.12"])
+      s.add_development_dependency(%q<test-unit>, [">= 2.1.1"])
+      s.add_development_dependency(%q<rake>, ["~> 0.9.2.2"])
+      s.add_development_dependency(%q<rdoc>, ["~> 3.12"])
+      s.add_development_dependency(%q<test-unit>, [">= 2.1.1"])
+      s.add_development_dependency(%q<rake>, ["~> 0.9.2.2"])
+      s.add_development_dependency(%q<rdoc>, ["~> 3.12"])
+      s.add_development_dependency(%q<test-unit>, [">= 2.1.1"])
+      s.add_development_dependency(%q<rake>, ["~> 0.9.2.2"])
+      s.add_development_dependency(%q<rdoc>, ["~> 3.12"])
       s.add_runtime_dependency(%q<mysql>, ["= 2.8.1"])
-      s.add_runtime_dependency(%q<pg>, ["= 0.9.0"])
+      s.add_runtime_dependency(%q<pg>, ["~> 0.11.0"])
       s.add_development_dependency(%q<test-unit>, [">= 2.1.1"])
+      s.add_development_dependency(%q<rake>, ["~> 0.9.2.2"])
+      s.add_development_dependency(%q<rdoc>, ["~> 3.12"])
     else
+      s.add_dependency(%q<mysql2psql>, [">= 0"])
+      s.add_dependency(%q<bundler>, ["~> 1.0.21"])
+      s.add_dependency(%q<jeweler>, ["~> 1.5.2"])
+      s.add_dependency(%q<bundler>, ["~> 1.0.21"])
+      s.add_dependency(%q<jeweler>, ["~> 1.5.2"])
+      s.add_dependency(%q<bundler>, ["~> 1.0.21"])
+      s.add_dependency(%q<jeweler>, ["~> 1.5.2"])
+      s.add_dependency(%q<test-unit>, [">= 2.1.1"])
+      s.add_dependency(%q<rake>, ["~> 0.9.2.2"])
+      s.add_dependency(%q<rdoc>, ["~> 3.12"])
+      s.add_dependency(%q<test-unit>, [">= 2.1.1"])
+      s.add_dependency(%q<rake>, ["~> 0.9.2.2"])
+      s.add_dependency(%q<rdoc>, ["~> 3.12"])
+      s.add_dependency(%q<test-unit>, [">= 2.1.1"])
+      s.add_dependency(%q<rake>, ["~> 0.9.2.2"])
+      s.add_dependency(%q<rdoc>, ["~> 3.12"])
+      s.add_dependency(%q<test-unit>, [">= 2.1.1"])
+      s.add_dependency(%q<rake>, ["~> 0.9.2.2"])
+      s.add_dependency(%q<rdoc>, ["~> 3.12"])
+      s.add_dependency(%q<test-unit>, [">= 2.1.1"])
+      s.add_dependency(%q<rake>, ["~> 0.9.2.2"])
+      s.add_dependency(%q<rdoc>, ["~> 3.12"])
       s.add_dependency(%q<mysql>, ["= 2.8.1"])
-      s.add_dependency(%q<pg>, ["= 0.9.0"])
+      s.add_dependency(%q<pg>, ["~> 0.11.0"])
       s.add_dependency(%q<test-unit>, [">= 2.1.1"])
+      s.add_dependency(%q<rake>, ["~> 0.9.2.2"])
+      s.add_dependency(%q<rdoc>, ["~> 3.12"])
     end
   else
+    s.add_dependency(%q<mysql2psql>, [">= 0"])
+    s.add_dependency(%q<bundler>, ["~> 1.0.21"])
+    s.add_dependency(%q<jeweler>, ["~> 1.5.2"])
+    s.add_dependency(%q<bundler>, ["~> 1.0.21"])
+    s.add_dependency(%q<jeweler>, ["~> 1.5.2"])
+    s.add_dependency(%q<bundler>, ["~> 1.0.21"])
+    s.add_dependency(%q<jeweler>, ["~> 1.5.2"])
+    s.add_dependency(%q<test-unit>, [">= 2.1.1"])
+    s.add_dependency(%q<rake>, ["~> 0.9.2.2"])
+    s.add_dependency(%q<rdoc>, ["~> 3.12"])
+    s.add_dependency(%q<test-unit>, [">= 2.1.1"])
+    s.add_dependency(%q<rake>, ["~> 0.9.2.2"])
+    s.add_dependency(%q<rdoc>, ["~> 3.12"])
+    s.add_dependency(%q<test-unit>, [">= 2.1.1"])
+    s.add_dependency(%q<rake>, ["~> 0.9.2.2"])
+    s.add_dependency(%q<rdoc>, ["~> 3.12"])
+    s.add_dependency(%q<test-unit>, [">= 2.1.1"])
+    s.add_dependency(%q<rake>, ["~> 0.9.2.2"])
+    s.add_dependency(%q<rdoc>, ["~> 3.12"])
+    s.add_dependency(%q<test-unit>, [">= 2.1.1"])
+    s.add_dependency(%q<rake>, ["~> 0.9.2.2"])
+    s.add_dependency(%q<rdoc>, ["~> 3.12"])
     s.add_dependency(%q<mysql>, ["= 2.8.1"])
-    s.add_dependency(%q<pg>, ["= 0.9.0"])
+    s.add_dependency(%q<pg>, ["~> 0.11.0"])
     s.add_dependency(%q<test-unit>, [">= 2.1.1"])
+    s.add_dependency(%q<rake>, ["~> 0.9.2.2"])
+    s.add_dependency(%q<rdoc>, ["~> 3.12"])
   end
 end
 

test/fixtures/config_all_options.yml

@@ -1,10 +1,10 @@
 mysql:
  hostname: localhost
  port: 3306
- socket: /tmp/mysql.sock
+ socket:
  username: somename
  password: secretpassword
- database: somename 
+ database: somename
 
 destination:
  # if file is given, output goes to file, else postgres
@@ -28,11 +28,23 @@ exclude_tables:
 - table5
 - table6
 
-# if supress_data is true, only the schema definition will be exported/migrated, and not the data
-supress_data: true
+# if suppress_data is true, only the schema definition will be exported/migrated, and not the data
+suppress_data: true
 
-# if supress_ddl is true, only the data will be exported/imported, and not the schema
-supress_ddl: false
+# if suppress_ddl is true, only the data will be exported/imported, and not the schema
+suppress_ddl: false
+
+# if suppress_sequence_update is true, the sequences for serial (auto-incrementing) columns
+# will not be update to the current maximum value of that column in the database
+# if suppress_ddl is not set to true, then this option is implied to be false as well (unless overridden here)
+suppress_sequence_update: false
+
+# if suppress_indexes is true, indexes will not be exported/migrated.
+suppress_indexes: false
 
 # if force_truncate is true, forces a table truncate before table loading
 force_truncate: false
+
+# if use_timezones is true, timestamp/time columns will be created in postgres as "with time zone"
+# rather than "without time zone"
+use_timezones: false

test/fixtures/seed_integration_tests.sql

@@ -4,21 +4,116 @@ DROP TABLE IF EXISTS numeric_types_basics;
 CREATE TABLE numeric_types_basics (
   id int,
 	f_tinyint TINYINT,
+	f_tinyint_u TINYINT UNSIGNED,
 	f_smallint SMALLINT,
+	f_smallint_u SMALLINT UNSIGNED,
 	f_mediumint MEDIUMINT,
+	f_mediumint_u MEDIUMINT UNSIGNED,
 	f_int INT,
+	f_int_u INT UNSIGNED,
 	f_integer INTEGER,
+	f_integer_u INTEGER UNSIGNED,
 	f_bigint BIGINT,
+	f_bigint_u BIGINT UNSIGNED,
 	f_real REAL,
 	f_double DOUBLE,
 	f_float FLOAT,
+	f_float_u FLOAT UNSIGNED,
 	f_decimal DECIMAL,
 	f_numeric NUMERIC
 );
 
 INSERT INTO numeric_types_basics VALUES
-(1,1,1,1,1,1,1,1,1,1,1,1),
-(2,2,2,2,2,2,2,2,2,2,2,2),
-(23,23,23,23,23,23,23,23,23,23,23,23);
+( 1, 2, 3, 4, 5, 6, 7, 8, 9,10,11,12,13,14,15,16,17,18,19),
+( 2, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1),
+( 3,23,23,23,23,23,23,23,23,23,23,23,23,23,23,23,23,23,23),
+( 4, -128,   0,-32768,     0,-8388608,        0,-2147483648,          0,-2147483648,          0,-9223372036854775808,                    0, 1, 1, 1, 1, 1, 1),
+( 5,  127, 255, 32767, 65535, 8388607, 16777215, 2147483647, 4294967295, 2147483647, 4294967295, 9223372036854775807, 18446744073709551615, 1, 1, 1, 1, 1, 1);
 
 
+
+DROP TABLE IF EXISTS basic_autoincrement;
+CREATE TABLE basic_autoincrement (
+  auto_id INT(11) NOT NULL AUTO_INCREMENT,
+	auto_dummy INT,
+	PRIMARY KEY (auto_id)
+);
+
+INSERT INTO basic_autoincrement(auto_dummy) VALUES
+(1),(2),(23);
+
+-- see GH#22 float conversion error
+DROP TABLE IF EXISTS numeric_type_floats;
+CREATE TABLE numeric_type_floats (
+ latitude FLOAT,
+ longitude FLOAT
+);
+
+INSERT INTO numeric_type_floats(latitude,longitude) VALUES
+(1.1,2.2);
+
+-- see GH#18 smallint error
+DROP TABLE IF EXISTS gh18_smallint;
+CREATE TABLE gh18_smallint (
+ s_smallint SMALLINT,
+ u_smallint SMALLINT UNSIGNED
+);
+
+INSERT INTO gh18_smallint(s_smallint,u_smallint) VALUES
+(-32768,32767),
+(-1,0),
+(32767,65535);
+
+-- see https://github.com/maxlapshin/mysql2postgres/issues/27
+DROP TABLE IF EXISTS test_boolean_conversion;
+CREATE TABLE test_boolean_conversion (
+  test_name VARCHAR(25),
+  bit_1 BIT(1),
+  tinyint_1 TINYINT(1),
+  bit_1_default_0 BIT(1) DEFAULT 0,
+  bit_1_default_1 BIT(1) DEFAULT 1,
+  tinyint_1_default_0 TINYINT(1) DEFAULT 0,
+  tinyint_1_default_1 TINYINT(1) DEFAULT 1,
+  tinyint_1_default_2 TINYINT(1) DEFAULT 2 -- Test the fact that 1 byte isn't limited to [0,1]
+);
+
+INSERT INTO test_boolean_conversion (test_name, bit_1, tinyint_1)
+VALUES ('test-null', NULL, NULL),
+       ('test-false', 0, 0),
+       ('test-true', 1, 1);
+INSERT INTO test_boolean_conversion (test_name, tinyint_1) VALUES ('test-true-nonzero', 2);
+
+CREATE OR REPLACE VIEW test_view AS
+SELECT b.test_name
+FROM test_boolean_conversion b;
+
+DROP TABLE IF EXISTS test_null_conversion;
+CREATE TABLE test_null_conversion (column_a VARCHAR(10));
+INSERT INTO test_null_conversion (column_a) VALUES (NULL);
+
+DROP TABLE IF EXISTS test_datetime_conversion;
+CREATE TABLE test_datetime_conversion (
+	column_a DATETIME,
+	column_b TIMESTAMP,
+	column_c DATETIME DEFAULT '0000-00-00',
+	column_d DATETIME DEFAULT '0000-00-00 00:00',
+	column_e DATETIME DEFAULT '0000-00-00 00:00:00',
+	column_f TIME
+);
+INSERT INTO test_datetime_conversion (column_a, column_f) VALUES ('0000-00-00 00:00', '08:15:30');
+
+DROP TABLE IF EXISTS test_index_conversion;
+CREATE TABLE test_index_conversion (column_a VARCHAR(10));
+CREATE UNIQUE INDEX index_test_index_conversion_on_column_a ON test_index_conversion (column_a);
+
+DROP TABLE IF EXISTS test_foreign_keys_child;
+DROP TABLE IF EXISTS test_foreign_keys_parent;
+CREATE TABLE test_foreign_keys_parent (id INT NOT NULL, PRIMARY KEY (id)) ENGINE=INNODB;
+CREATE TABLE test_foreign_keys_child (id INT, test_foreign_keys_parent_id INT,
+	INDEX par_ind (test_foreign_keys_parent_id),
+	FOREIGN KEY (test_foreign_keys_parent_id) REFERENCES test_foreign_keys_parent(id) ON DELETE CASCADE
+) ENGINE=INNODB;
+
+DROP TABLE IF EXISTS test_enum;
+CREATE TABLE test_enum (name ENUM('small', 'medium', 'large'));
+INSERT INTO test_enum (name) VALUES ('medium');

test/integration/convert_to_db_test.rb

@@ -4,26 +4,134 @@
 
 class ConvertToDbTest < Test::Unit::TestCase
 
-  class << self
-    def startup
-      seed_test_database
-      @@options=get_test_config_by_label(:localmysql_to_db_convert_all)
-      @@mysql2psql = Mysql2psql.new([@@options.filepath])
-      @@mysql2psql.convert
-      @@mysql2psql.writer.open
-    end
-    def shutdown
-      @@mysql2psql.writer.close
-      delete_files_for_test_config(@@options)
-    end
-  end
   def setup
+    $stdout = StringIO.new
+    $stderr = StringIO.new
+
+    seed_test_database
+    @options=get_test_config_by_label(:localmysql_to_db_convert_all)
+    @mysql2psql = Mysql2psql.new([@options.filepath])
+    @mysql2psql.convert
+    @mysql2psql.writer.open
   end
+
   def teardown
+    @mysql2psql.writer.close
+    delete_files_for_test_config(@options)
+
+    $stdout = STDOUT
+    $stderr = STDERR
+  end
+
+  def exec_sql_on_psql(sql, parameters=nil)
+    @mysql2psql.writer.conn.exec(sql, parameters)
+  end
+
+  def get_boolean_test_record(name)
+    exec_sql_on_psql('SELECT * FROM test_boolean_conversion WHERE test_name = $1', [name]).first
   end
 
   def test_table_creation
-    assert_true @@mysql2psql.writer.exists?('numeric_types_basics')
+    assert_true @mysql2psql.writer.exists?('numeric_types_basics')
+    assert_true @mysql2psql.writer.exists?('basic_autoincrement')
+    assert_true @mysql2psql.writer.exists?('numeric_type_floats')
+  end
+
+  def test_boolean_conversion_to_true
+    true_record = get_boolean_test_record('test-true')
+    assert_equal 't', true_record['bit_1']
+    assert_equal 't', true_record['tinyint_1']
+
+    true_nonzero_record = get_boolean_test_record('test-true-nonzero')
+    assert_equal 't', true_nonzero_record['tinyint_1']
   end
 
+  def test_boolean_conversion_to_false
+    false_record = get_boolean_test_record('test-false')
+    assert_equal 'f', false_record['bit_1']
+    assert_equal 'f', false_record['tinyint_1']
+  end
+
+  def test_boolean_conversion_of_null
+    null_record = get_boolean_test_record('test-null')
+    assert_nil null_record['bit_1']
+    assert_nil null_record['tinyint_1']
+  end
+
+  def test_null_conversion
+    result = exec_sql_on_psql('SELECT column_a FROM test_null_conversion').first
+    assert_nil result['column_a']
+  end
+
+  def test_datetime_conversion
+    result = exec_sql_on_psql('SELECT column_a, column_f FROM test_datetime_conversion').first
+    assert_equal '1970-01-01 00:00:00', result['column_a']
+    assert_equal '08:15:30', result['column_f']
+  end
+
+  def test_datetime_defaults
+    result = exec_sql_on_psql(<<-SQL)
+      SELECT a.attname,
+        pg_catalog.format_type(a.atttypid, a.atttypmod),
+        (SELECT substring(pg_catalog.pg_get_expr(d.adbin, d.adrelid) for 128)
+         FROM pg_catalog.pg_attrdef d
+         WHERE d.adrelid = a.attrelid AND d.adnum = a.attnum AND a.atthasdef) AS default
+      FROM pg_catalog.pg_attribute a
+      WHERE a.attrelid = 'test_datetime_conversion'::regclass AND a.attnum > 0
+    SQL
+
+    assert_equal 6, result.count
+
+    result.each do |row|
+      if row["attname"] == "column_f"
+        assert_equal "time without time zone", row["format_type"]
+      else
+        assert_equal "timestamp without time zone", row["format_type"]
+      end
+
+      case row["attname"]
+      when "column_a"
+        assert_nil row["default"]
+      when "column_b"
+        assert_equal "now()", row["default"]
+      when "column_c", "column_d", "column_e"
+        assert_equal "'1970-01-01 00:00:00'::timestamp without time zone", row["default"]
+      end
+    end
+  end
+
+  def test_index_conversion
+    result = exec_sql_on_psql(%(
+       SELECT pg_get_indexdef(indexrelid)
+       FROM pg_index
+       join pg_class on pg_class.oid=pg_index.indrelid
+       where pg_class.relname='test_index_conversion'
+    )).first
+    assert_equal "CREATE UNIQUE INDEX index_test_index_conversion_on_column_a ON test_index_conversion USING btree (column_a)", result["pg_get_indexdef"]
+  end
+
+  def test_foreign_keys
+    result = exec_sql_on_psql("SELECT conname, pg_catalog.pg_get_constraintdef(r.oid, true) as condef FROM pg_catalog.pg_constraint r WHERE r.conrelid = 'test_foreign_keys_child'::regclass")
+    expected = {"condef" => "FOREIGN KEY (test_foreign_keys_parent_id) REFERENCES test_foreign_keys_parent(id) ON UPDATE RESTRICT ON DELETE CASCADE", "conname" => "test_foreign_keys_child_test_foreign_keys_parent_id_fkey"}
+    assert_equal expected, result.first
+  end
+
+  def test_output
+    $stdout.rewind
+    actual = $stdout.read
+
+    assert_match /Counting rows of test_foreign_keys_child/, actual
+  end
+
+  def test_enum
+    result = exec_sql_on_psql(<<-SQL)
+      SELECT r.conname, pg_catalog.pg_get_constraintdef(r.oid, true)
+      FROM pg_catalog.pg_constraint r
+      WHERE r.conrelid = 'test_enum'::regclass AND r.contype = 'c'
+      ORDER BY 1
+    SQL
+
+    assert_equal 1, result.count
+    assert_equal "CHECK (name::text = ANY (ARRAY['small'::character varying, 'medium'::character varying, 'large'::character varying]::text[]))", result.first["pg_get_constraintdef"]
+  end
 end

test/integration/convert_to_file_test.rb

@@ -4,63 +4,118 @@
 
 class ConvertToFileTest < Test::Unit::TestCase
 
-  class << self
-    def startup
-      seed_test_database
-      @@options=get_test_config_by_label(:localmysql_to_file_convert_all)
-      @@mysql2psql = Mysql2psql.new([@@options.filepath])
-      @@mysql2psql.convert
-      @@content = IO.read(@@mysql2psql.options.destfile)
-    end
-    def shutdown
-      delete_files_for_test_config(@@options)
-    end
-  end
   def setup
+    seed_test_database
+    @options=get_test_config_by_label(:localmysql_to_file_convert_all)
+    @mysql2psql = Mysql2psql.new([@options.filepath])
+    @mysql2psql.convert
+    @content = IO.read(@mysql2psql.options.destfile)
   end
   def teardown
+    delete_files_for_test_config(@options)
   end
   def content
-    @@content
+    @content
   end
-
+
+  # verify table creation
   def test_table_creation
     assert_not_nil content.match('DROP TABLE IF EXISTS "numeric_types_basics" CASCADE')
     assert_not_nil content.match(/CREATE TABLE "numeric_types_basics"/)
   end
 
+  # tests for the conversion of numeric types
+  def get_basic_numerics_match(column)
+    Regexp.new('CREATE TABLE "numeric_types_basics".*"' + column + '" ([^\n]*).*\)', Regexp::MULTILINE).match( content )
+  end
   def test_basic_numerics_tinyint
-    assert_not_nil Regexp.new('CREATE TABLE "numeric_types_basics".*"f_tinyint" smallint,.*\)', Regexp::MULTILINE).match( content )
+    match = get_basic_numerics_match( 'f_tinyint' )
+    assert_match /smallint/,match[1]
+  end
+  def test_basic_numerics_tinyint_u
+    match = get_basic_numerics_match( 'f_tinyint_u' )
+    assert_match /smallint/,match[1]
   end
   def test_basic_numerics_smallint
-    assert_not_nil Regexp.new('CREATE TABLE "numeric_types_basics".*"f_smallint" integer,.*\)', Regexp::MULTILINE).match( content )
+    match = get_basic_numerics_match( 'f_smallint' )
+    assert_match /smallint/,match[1]
+  end
+  def test_basic_numerics_smallint_u
+    match = get_basic_numerics_match( 'f_smallint_u' )
+    assert_match /integer/,match[1]
   end
   def test_basic_numerics_mediumint
-    assert_not_nil Regexp.new('CREATE TABLE "numeric_types_basics".*"f_mediumint" integer,.*\)', Regexp::MULTILINE).match( content )
+    match = get_basic_numerics_match( 'f_mediumint' )
+    assert_match /integer/,match[1]
   end
   def test_basic_numerics_int
-    assert_not_nil Regexp.new('CREATE TABLE "numeric_types_basics".*"f_int" integer,.*\)', Regexp::MULTILINE).match( content )
+    match = get_basic_numerics_match( 'f_int' )
+    assert_match /integer/,match[1]
   end
   def test_basic_numerics_integer
-    assert_not_nil Regexp.new('CREATE TABLE "numeric_types_basics".*"f_integer" integer,.*\)', Regexp::MULTILINE).match( content )
+    match = get_basic_numerics_match( 'f_integer' )
+    assert_match /integer/,match[1]
   end
   def test_basic_numerics_bigint
-    assert_not_nil Regexp.new('CREATE TABLE "numeric_types_basics".*"f_bigint" bigint,.*\)', Regexp::MULTILINE).match( content )
+    match = get_basic_numerics_match( 'f_bigint' )
+    assert_match /bigint/,match[1]
+    #assert_not_nil Regexp.new('CREATE TABLE "numeric_types_basics".*"f_bigint" bigint,.*\)', Regexp::MULTILINE).match( content )
   end
   def test_basic_numerics_real
-    assert_not_nil Regexp.new('CREATE TABLE "numeric_types_basics".*"f_real" double precision,.*\)', Regexp::MULTILINE).match( content )
+    match = get_basic_numerics_match( 'f_real' )
+    assert_match /double precision/,match[1]
+    #assert_not_nil Regexp.new('CREATE TABLE "numeric_types_basics".*"f_real" double precision,.*\)', Regexp::MULTILINE).match( content )
   end
   def test_basic_numerics_double
-    assert_not_nil Regexp.new('CREATE TABLE "numeric_types_basics".*"f_double" double precision,.*\)', Regexp::MULTILINE).match( content )
+    match = get_basic_numerics_match( 'f_double' )
+    assert_match /double precision/,match[1]
+    #assert_not_nil Regexp.new('CREATE TABLE "numeric_types_basics".*"f_double" double precision,.*\)', Regexp::MULTILINE).match( content )
   end
   def test_basic_numerics_float
-    assert_not_nil Regexp.new('CREATE TABLE "numeric_types_basics".*"f_float" numeric\(20, 0\),.*\)', Regexp::MULTILINE).match( content )
+    match = get_basic_numerics_match( 'f_float' )
+    assert_match /double precision/,match[1]
+    #assert_not_nil Regexp.new('CREATE TABLE "numeric_types_basics".*"f_float" double precision.*\)', Regexp::MULTILINE).match( content )
+  end
+  def test_basic_numerics_float_u
+    match = get_basic_numerics_match( 'f_float_u' )
+    assert_match /double precision/,match[1]
+    #assert_not_nil Regexp.new('CREATE TABLE "numeric_types_basics".*"f_float_u" double precision.*\)', Regexp::MULTILINE).match( content )
   end
   def test_basic_numerics_decimal
-    assert_not_nil Regexp.new('CREATE TABLE "numeric_types_basics".*"f_decimal" numeric\(10, 0\),.*\)', Regexp::MULTILINE).match( content )
+    match = get_basic_numerics_match( 'f_decimal' )
+    assert_match /numeric/,match[1]
+    #assert_not_nil Regexp.new('CREATE TABLE "numeric_types_basics".*"f_decimal" numeric\(10, 0\),.*\)', Regexp::MULTILINE).match( content )
   end
   def test_basic_numerics_numeric
-    assert_not_nil Regexp.new('CREATE TABLE "numeric_types_basics".*"f_numeric" numeric\(10, 0\)[\w\n]*\)', Regexp::MULTILINE).match( content )
+    match = get_basic_numerics_match( 'f_numeric' )
+    assert_match /numeric/,match[1]
+    #assert_not_nil Regexp.new('CREATE TABLE "numeric_types_basics".*"f_numeric" numeric\(10, 0\)[\w\n]*\)', Regexp::MULTILINE).match( content )
   end
 
+  # test boolean conversion
+  def get_boolean_column_definition_match(column)
+    Regexp.new('CREATE TABLE "test_boolean_conversion".*"' + column + '" ([^\n]*)[^;]*', Regexp::MULTILINE).match(content)[1]
+  end
+  def test_boolean_default_values
+    assert_match /DEFAULT false/, get_boolean_column_definition_match('bit_1_default_0')
+    assert_match /DEFAULT false/, get_boolean_column_definition_match('tinyint_1_default_0')
+
+    assert_match /DEFAULT true/, get_boolean_column_definition_match('bit_1_default_1')
+    assert_match /DEFAULT true/, get_boolean_column_definition_match('tinyint_1_default_1')
+    assert_match /DEFAULT true/, get_boolean_column_definition_match('tinyint_1_default_2')
+  end
+
+  # test autoincrement handling
+  def test_autoincrement
+    assert_not_nil Regexp.new('CREATE TABLE "basic_autoincrement".*"auto_id" integer DEFAULT.*\)', Regexp::MULTILINE).match( content )
+  end
+
+  # test view creation (or lack thereof!)
+  def test_should_not_copy_views_as_tables
+    assert_no_match /CREATE TABLE "test_view"/, content
+  end
+
+  def test_truncate
+    assert_match /TRUNCATE/, content
+  end
 end

test/integration/converter_test.rb

@@ -4,18 +4,12 @@
 
 class ConverterTest < Test::Unit::TestCase
 
-  class << self
-    def startup
-      seed_test_database
-      @@options=get_test_config_by_label(:localmysql_to_file_convert_nothing)
-    end
-    def shutdown
-      delete_files_for_test_config(@@options)
-    end
-  end
   def setup
+    seed_test_database
+    @@options=get_test_config_by_label(:localmysql_to_file_convert_nothing)
   end
   def teardown
+    delete_files_for_test_config(@@options)
   end
   def options
     @@options

test/integration/mysql_reader_base_test.rb

@@ -4,39 +4,31 @@
 
 class MysqlReaderBaseTest < Test::Unit::TestCase
 
-  class << self
-    def startup
-      seed_test_database
-      @@options = get_test_config_by_label(:localmysql_to_file_convert_nothing)
-    end
-    def shutdown
-      delete_files_for_test_config(@@options)
-    end
-  end
   def setup
+    seed_test_database
+    @options = get_test_config_by_label(:localmysql_to_file_convert_nothing)
   end
+
   def teardown
-  end
-  def options
-    @@options
+    delete_files_for_test_config(@options)
   end
 
   def test_mysql_connection
     assert_nothing_raised do
-      reader = Mysql2psql::MysqlReader.new(options)
+      reader = Mysql2psql::MysqlReader.new(@options)
     end
   end
   def test_mysql_reconnect
     assert_nothing_raised do
-      reader = Mysql2psql::MysqlReader.new(options)
+      reader = Mysql2psql::MysqlReader.new(@options)
       reader.reconnect
     end
   end
   def test_mysql_connection_without_port
     assert_nothing_raised do
-      options.mysqlport = ""
-      options.mysqlsocket = ""
-      reader = Mysql2psql::MysqlReader.new(options)
+      @options.mysqlport = ""
+      @options.mysqlsocket = ""
+      reader = Mysql2psql::MysqlReader.new(@options)
     end
   end
 end

test/integration/mysql_reader_test.rb

@@ -2,22 +2,16 @@
 
 class MysqlReaderTest < Test::Unit::TestCase
 
-  class << self
-    def startup
-      seed_test_database
-      @@options = get_test_config_by_label(:localmysql_to_file_convert_nothing)
-      @@reader=get_test_reader(@@options)
-    end
-    def shutdown
-      delete_files_for_test_config(@@options)
-    end
-  end
   def setup
+    seed_test_database
+    @options = get_test_config_by_label(:localmysql_to_file_convert_nothing)
+    @reader=get_test_reader(@options)
   end
   def teardown
+    delete_files_for_test_config(@options)
   end
   def reader
-    @@reader
+    @reader
   end
 
   def test_db_connection
@@ -31,7 +25,7 @@ def test_tables_collection
     assert_equal 'numeric_types_basics', values[0].name
   end
   def test_paginated_read
-    expected_rows=3
+    expected_rows=5
     page_size=2
     expected_pages=(1.0 * expected_rows / page_size).ceil
 

test/integration/postgres_db_writer_base_test.rb

@@ -4,26 +4,18 @@
 
 class PostgresDbWriterBaseTest < Test::Unit::TestCase
 
-  class << self
-    def startup
-      seed_test_database
-      @@options=get_test_config_by_label(:localmysql_to_db_convert_nothing)
-    end
-    def shutdown
-      delete_files_for_test_config(@@options)
-    end
-  end
   def setup
+    seed_test_database
+    @options = get_test_config_by_label(:localmysql_to_db_convert_nothing)
   end
+
   def teardown
-  end
-  def options
-    @@options
+    delete_files_for_test_config(@options)
   end
 
   def test_pg_connection
     assert_nothing_raised do
-      reader = Mysql2psql::PostgresDbWriter.new(options)
+      reader = Mysql2psql::PostgresDbWriter.new(@options)
     end
   end
 

test/lib/ext_test_unit.rb

@@ -26,5 +26,8 @@ def assert_false(object, message="")
   def assert_true(object, message="")
     assert_equal(true, object, message)
   end
+  def assert_nil(object, message="")
+    assert_equal(nil, object, message)
+  end
 end
 

test/lib/test_helper.rb

@@ -12,7 +12,8 @@
 def seed_test_database
   options=get_test_config_by_label(:localmysql_to_file_convert_nothing)
   seedfilepath = "#{File.dirname(__FILE__)}/../fixtures/seed_integration_tests.sql"
-  rc=system("mysql -u#{options.mysqlusername} #{options.mysqldatabase} < #{seedfilepath}")
+  mysql_cmd = `which mysql`.empty? ? 'mysql5' : 'mysql'
+  rc=system("#{mysql_cmd} -u#{options.mysqlusername} #{options.mysqldatabase} < #{seedfilepath}")
   raise StandardError unless rc
   return true
 rescue
@@ -25,7 +26,7 @@ def get_test_reader(options)
   require 'mysql2psql/mysql_reader'
   Mysql2psql::MysqlReader.new(options)
 rescue
-  raise StandardError.new("Failed to initialize integration test db. See README for setup requirements.")  
+  raise StandardError.new("Failed to initialize integration test db. See README for setup requirements.")
 end
 
 def get_test_file_writer(options)
@@ -53,12 +54,13 @@ def get_temp_file(basename)
   path
 end
 
-
-def get_new_test_config(to_file = true, include_tables = [], exclude_tables = [], supress_data = false, supress_ddl = false, force_truncate = false)
+def get_new_test_config(options={})
   require 'mysql2psql/config'
   require 'mysql2psql/config_base'
-  to_filename = to_file ? get_temp_file('mysql2psql_tmp_output') : nil
-  configtext = Mysql2psql::Config.template(to_filename, include_tables, exclude_tables, supress_data, supress_ddl, force_truncate)
+  if options.delete(:to_file)
+    options[:to_filename] = get_temp_file('mysql2psql_tmp_output')
+  end
+  configtext = Mysql2psql::Config.template(options)
   configfile=get_temp_file('mysql2psql_tmp_config')
   File.open(configfile, 'w') {|f| f.write(configtext) }
   Mysql2psql::ConfigBase.new( configfile )
@@ -67,18 +69,59 @@ def get_new_test_config(to_file = true, include_tables = [], exclude_tables = []
 end
 
 def get_test_config_by_label(name)
-  case name
+  options = case name
   when :localmysql_to_file_convert_nothing
-    get_new_test_config(true, ['unobtainium'], ['kryptonite'], true, true, false)
+    {
+      :to_file => true,
+      :include_tables => ['unobtainium'],
+      :exclude_tables => ['kryptonite'],
+      :suppress_data => true,
+      :suppress_ddl => true,
+      :supress_sequence_update => false,
+      :suppress_indexes => false,
+      :force_truncate => false,
+      :use_timezones => false
+    }
   when :localmysql_to_file_convert_all
-    get_new_test_config(true, [], [], false, false, false)
+    {
+      :to_file => true,
+      :include_tables => [],
+      :exclude_tables => [],
+      :suppress_data => false,
+      :suppress_ddl => false,
+      :supress_sequence_update => false,
+      :suppress_indexes => false,
+      :force_truncate => true,
+      :use_timezones => false
+    }
   when :localmysql_to_db_convert_all
-    get_new_test_config(false, [], [], false, false, false)
+    {
+      :to_file => false,
+      :include_tables => [],
+      :exclude_tables => [],
+      :suppress_data => false,
+      :suppress_ddl => false,
+      :supress_sequence_update => false,
+      :suppress_indexes => false,
+      :force_truncate => false,
+      :use_timezones => false
+    }
   when :localmysql_to_db_convert_nothing
-    get_new_test_config(false, ['unobtainium'], ['kryptonite'], true, true, false)
+    {
+      :to_file => false,
+      :include_tables => ['unobtainium'],
+      :exclude_tables => ['kryptonite'],
+      :suppress_data => true,
+      :suppress_ddl => true,
+      :supress_sequence_update => false,
+      :suppress_indexes => false,
+      :force_truncate => false,
+      :use_timezones => false
+    }
   else
     raise StandardError.new("Invalid label: #{name}")
   end
+  get_new_test_config(options)
 end
 
 def delete_files_for_test_config(config)

test/units/config_test.rb

@@ -23,9 +23,51 @@ def test_config_file_not_found
       value = Mysql2psql::Config.new(configfile_not_found, false)
     end
   end
+
   def test_initialize_new_config_file
     assert_raise(Mysql2psql::ConfigurationFileInitialized) do
       value = Mysql2psql::Config.new(configfile_new, true)
     end
   end
+
+  def test_config_option_pgdatabase_as_array_index
+    expected = 'somename'
+    config = Mysql2psql::Config.new(configfile_all_opts, false)
+    assert_equal expected,config[:pgdatabase]
+  end
+  def test_template_option_to_filename
+    expected = 'test_filename'
+    value = Mysql2psql::Config.template({ :to_filename => expected })
+    assert_match /file: #{expected}/,value
+  end
+  def test_template_option_suppress_data
+    expected = true
+    value = Mysql2psql::Config.template({ :suppress_data => expected })
+    assert_match /suppress_data: #{expected}/,value #NB: option spelling needs fixing
+  end
+  def test_template_option_suppress_ddl
+    expected = true
+    value = Mysql2psql::Config.template({ :suppress_ddl => expected })
+    assert_match /suppress_ddl: #{expected}/,value #NB: option spelling needs fixing
+  end
+  def test_template_option_suppress_sequence_update
+    expected = true
+    value = Mysql2psql::Config.template({ :suppress_sequence_update => expected })
+    assert_match /suppress_sequence_update: #{expected}/,value #NB: option spelling needs fixing
+  end
+  def test_template_option_suppress_indexes
+    expected = true
+    value = Mysql2psql::Config.template({ :suppress_indexes => expected })
+    assert_match /suppress_indexes: #{expected}/,value #NB: option spelling needs fixing
+  end
+  def test_template_option_force_truncate
+    expected = true
+    value = Mysql2psql::Config.template({ :force_truncate => expected })
+    assert_match /force_truncate: #{expected}/,value
+  end
+  def test_template_option_use_timezones
+    expected = true
+    value = Mysql2psql::Config.template({ :use_timezones => expected })
+    assert_match /use_timezones: #{expected}/,value
+  end
 end