I’m wearing down pretty quickly. Mandatory overtime isn’t enjoyable. It’s coming at me steady and fast. I’m told I’ll have help. I’m told they’ll hire another person. That extra help never comes. Nothing gets easier. I just fall further behind. It’s tiring. I’m bored with it. I’m ready to move on to bigger and better things. I’m ready to get my life moving in a direction I want it to move.

Total Palestinian children killed at the hands of Israeli police forces between 2000-2014:
1,397

Total US children murdered in the same time span:
7,437

What the flying fuck, people? How is this acceptable in any way? Let’s look at more numbers, shall we?

4.44 million people. 0.03% of the ENTIRE population.

My current tweet dumper iteration dumps everything collected into a flat file. That’s fine for intermittent use, but won’t do much good when we eventually get into the “big leagues” and start grabbing at streams of data that will include above and beyond 3200 tweets. Flat files will quickly swell to sizes that are no longer manageable.

Wat do.

Well, there are hundreds of “database backend” options. Literally hundreds. We’re inevitably going to be storing tweets from various sources with multiple goals in mind. A veritable forest of *.csv files doesn’t neatly organize our data. SQLite3 will provide the backend. It’s local, zero configuration required, and we can concentrate all our information into a single file with multiple tables.

I’ve made improvements to the tweet dumper. Added geo information. Also added progress bar, so check gem requirements.

#!/bin/env ruby
# encoding: utf-8

require 'twitter'
require 'csv'
require 'progressbar'

client = Twitter::REST::Client.new do |config|
	config.consumer_key = "YoursHere"
	config.consumer_secret = "YoursHere"
	config.access_token = "YoursHere"
	config.access_token_secret = "YoursHere"
end

scrname = String.new ARGV[0]

def collect_with_max_id(collection=[], max_id=nil, &block)
  response = yield(max_id)
  collection += response
  response.empty? ? collection.flatten : collect_with_max_id(collection, response.last.id - 1, &block)
end

def client.get_all_tweets(user)
  twtcount = user(user).statuses_count
  if twtcount > 3200
      twtcount = 3200 / 200
  else
      twtcount = twtcount / 200
  end
  pbar = ProgressBar.new("Downloading", twtcount)
  collect_with_max_id do |max_id|
    pbar.inc
    options = {:count => 200, :include_rts => true}
    options[:max_id] = max_id unless max_id.nil?
    user_timeline(user, options)
  end
end

junk = client.get_all_tweets(scrname)

CSV.open("#{scrname}.csv", "w") do |csv|
	junk.each do |tweet|
		csv << [tweet.id, tweet.created_at, tweet.user.screen_name, tweet.text, tweet.source, tweet.geo]
	end
end

I don’t comment. Sorry. I guess I can go back through and comment where it’s helpful and repost another time. It works with Ruby 1.9.2 anyways.

Requires the twitter gem. Install it as per usual. Code as follows:

#!/bin/env ruby
# encoding: utf-8

require 'twitter'
require 'csv'

client = Twitter::REST::Client.new do |config|
	config.consumer_key = "insert"
	config.consumer_secret = "insert"
	config.access_token = "insert"
	config.access_token_secret = "insert"
end

def collect_with_max_id(collection=[], max_id=nil, &block)
  response = yield(max_id)
  collection += response
  response.empty? ? collection.flatten : collect_with_max_id(collection, response.last.id - 1, &block)
end

def client.get_all_tweets(user)
  collect_with_max_id do |max_id|
    options = {:count => 200, :include_rts => true}
    options[:max_id] = max_id unless max_id.nil?
    user_timeline(user, options)
  end
end

junk = client.get_all_tweets(ARGV[0])

CSV.open("#{ARGV[0]}.csv", "w") do |csv|
	junk.each do |tweet|
		csv << [tweet.id, tweet.created_at, tweet.user.screen_name, tweet.text, tweet.source, tweet.geo]
	end
end

Excellent. I’m going to revise it as necessary, but it’s a most effective scraper. Though I’d love to add some sort of progress bar to it, haven’t succeeded in that yet. I’ll keep you posted and update it as the iterations of this thing change. It was smashed together from the twitter gem’s bare scraper and CSV output added. I’m quite pleased. Going to also consider adding time and date statistics compilation. I might just write an entirely separate script for that. Not sure yet.

This is pretty excellent. Check it out.

It’s called “Envy Code R”, you can grab it here.

I might use this as my “hosted font of choice” for rendering scripts and the like. It’s pretty damn cool.

If you’re in the automotive manufacture industry, you’re pretty familiar with re-certification. It’s that time of year everyone panics and freaks the fuck out about either surveillance or re-cert audits, works seven days a week, ten hour days, and has a mild heart attack over every misspelling or problematic procedure in the control documents for their company.

What disturbs me is that I absolutely despise this process. I hate it, yet no matter where I go I won’t be able to avoid it. I need to get out of the department that has to deal with that garbage. I need to get ietnto engineering, production, some other department. It doesn’t matter. I can’t put in these hours. It’s too much. I have a family. I have a life outside of work. It’s effecting me in some rather deeply disturbing ways. I don’t sleep as much as I need to and every day feels like a hallucination.

Jojo’s PO contact info via @5hm00p

Attention to everyone who was harassed or threatened by Joseph Camp:

I just got word that his sentencing is Tuesday, and he's going to he
served 5 months and when he's out he's free to do as he pleases. I
myself don't believe this is someone who belongs in a normal functioning
society, therefore a few of us have started a campaign to get him some
sort of supervision when he is released. How can you help?
Simple....write to:

US Parole Officer George Martin
100 State St.
Rochester, NY 14614

or

you can reach him by phone at (585) 666-5901


and tell him about your encounter with this maniac. The last thing we
want is him around again harassing people for Sue Basko (don't worry,
disbar comin) again and invading people's lives and their privacy. This
is your chance to put a completely horrible human being away and
hopefully make him learn his lesson.