Does your ping look like this when using Android’s sweet USB tethering feature? I tried lots of things:

• Disabling/Enabling USB Tethering
• Disconnecting/Reconnecting the USB cable
• Restarting the phone
• Disabling/Enabling the network interface on the computer
• Restarting the computer

All of this got me exactly no where. The phone is still connected to the interwho, but it’s just not sharing it with my laptop. So sad. So, thinking a little bit more laterally (read as, poking around looking for a sign from the gods), I tried toggling the data access of the phone. This means going to Menu > Wireless & networks > Mobile network settings > Data enabled and unchecking the box.

Why does it make sense to turn off data when what you want is data? Well… it doesn’t. But, if you then turn the data back on, the phone may well realize that it has been fraudulently denying your poor computer all the data goodness the marvelous (if mystifying) Android platform promises.  I say “may well” because with the fragmentation of the Android market and the amount of irresponsible hackery the carriers subject the OS to, this may not actually fix your problem though it fixes mine. I’m using a vanilla-tainted-by-T-Mobile-and-LG version of Android 2.3 running on the T-Mobile G2X. This phone also goes by the name of Optimus 2X. Soon enough I’ll ditch it for the Galaxy Nexus, but I digress.

This what I get trying to ping after turning back on the “Enable Data” option. Hooray. All it took was doing an action equivalently counterintuitive to the famous “Oh you want to shut down? First, go to start…” paradigm. At least this one is unintentional.

Also, you may be wondering why on earth I would be using USB Tethering when my phone provides WiFi Hotspot capability. The answer is that I’ve found the latency to be lower and the signal to be more consistent using USB Tethering. The bandwidth is also higher, which I suppose makes sense because one phone using the same wireless antenna for sending and receiving data probably isn’t sending as fast as it could be. I should be careful to specify that I don’t actually know that is what’s happening. Anyway, USB Tethering gives me a faster connection. The other perk is that my laptop, which I’m going to plug in when I get to work, can power the phone while it’s on USB Tethering. With WiFi Hotspotting, I’m pretty guaranteed to drain my phone to the point where I will have to plug it in to make it through the day. There you have it.

If you’ve never used macros in any context, you may not have a sense of how powerful a tool they really are. Macros give you the ability to take a series of actions that achieve some goal and put them into a recording. You can then replay those actions again and again. This means that complex transformations on some text only have to be done correctly one time. The instructions can be captured and reused. We humans are pretty good at getting something hard done correctly once. Our error rate goes up the more times we do the thing. So, macros are helpful because when you get the steps right once, you get to use your own actions again and again. The replay is exact. The computer doesn’t accidentaly skip step three like you or I might the fifteenth time we walk through the process.

The other benefit is that the steps get executed at human speed (i.e., slow and clumsy speed) only the first time. Each time the actions are replayed, all the time you spent head scratching and double checking that you’re doing the right thing disappears and the actions get executed as fast as the computer can do it. Vim has macros built-in as do a lot of applications. Oh how I long for real life macrobot. Imagine being able to program a robot, one time, to fold laundry. I would never match a pair of socks again. Baby steps.

The background for all this macro talk is a task I took on recently. I needed to get the data out of a bunch of hand-rolled, inconsistently formatted PDFs and into some reasonable format. The goal was to have a structured data representation of the data that could be easily maintained over time. I guess it’s not strictly insane to try to keep a bunch of willy-nilly PDF files full of timetables up to date, but it’s far from efficient. To achieve this textual cleaning of the stables, I wanted to use Vim Macros because I knew they were a good tool for the job and I didn’t already know how to use them. Before this little jaunt, my exposure to Vim’s macros was pretty much just accidentally starting the recording process and trying every combination of colon, escape, and q to get out of it. Like I said, I aspire to be a competent Vim user. Baby steps.

Anyway, my tactic was to copy and paste the tables from the PDF files into a Vim document and determine an automated way to turn the data into a CSV format. Okay, actually I’m lying. I started by pasting that data into TextMate and trying to use find and replace hacks in addition to my amazing repetitive typing abilities to get the data into a CSV format. As I started to get into the rhythm of it, I could feel my inner smarty pants engineer sighing at me. Yes, I know there is no good reason to be using my pitiful human brain for simple, repeated tasks, but I don’t already know how to use them.  Yes, I should use the generic tools available for automating the process so I can get it right and do it fast, but I just want to get it done. This is a classic dilemma. Option one: Do it the slow, hard way that is guaranteed to get finished in a fixed amount of time but may introduce errors at worst and at best is a pain in the ass. Option two: Maybe do it the super fast way, assuming you can actually figure out how to use the scary new tools properly. XKCD 974 comes to mind.

After a few moments of complete philosophical system lock, I decided it was time to brain-up and make the computer do my bidding. Since I knew I needed to learn macros, I decided that would be the weapon with which I would dispatch these gobs of poorly structured nonsense.

Copying and pasting from the PDF, I get pretty garbled data, like this:

9:00 AM Location 1 6:30 PM Location 5
9:05 AM Location 2 6:35 PM Location 4
9:20 AM Location 3 7:45 PM Location 6
10:25 AM Location 4 7:55 PM Location 2
10:30 AM Location 5 8:00 PM Location 1
9:30 AM Location 1 7:30 PM Location 5
9:35 AM Location 2 7:35 PM Location 4
9:50 AM Location 3 8:45 PM Location 6
10:55 AM Location 4 8:55 PM Location 2
11:00 AM Location 5 9:00 PM Location 1
10:15 AM Location 1 8:10 PM Location 5
10:20 AM Location 2 8:15 PM Location 4
10:35 AM Location 3 9:25 PM Location 6
11:35 AM Location 4 9:35 PM Location 2
11:40 AM Location 5 9:40 PM Location 1
...and so on for about a hundred lines.

Though it’s bad, it’s a bearable starting place because it’s consistent in at least one way. For every time, the location that follows it is correct. The ideal CSV output would have the locations as the first row and the times for each location in the correct CSV column on the following rows.  Of course there are some hurdles here. In the PDF paste, there are two columns because the PDF was in a two column layout and whoever created it decided to just stretch one table across the page rather than created two separate tables. So, step one is to get the columns cleaned up. Easy enough, right? After all, I know VIM MACROS!!!

It’s macro time.

I actually didn’t know much about macros in Vim, but I’ve used them in other contexts. The basic idea is that a macro is a recording of some actions that can be replayed. There are some basic rules one has to know to get full value out of macros. As far as I can tell, there are two:

Remember the computer has no intuition so it must be told exactly what to do

Maybe this is obvious, but it’s worth reiterating that when you, as a human, say “select this text and do this with it” in your head, you’re glossing over tons and tons of steps that a computer will have to be told. To write a macro, you have to break down complex actions into each of the sub-actions until you arrive at steps simple enough for the computer to do it. Baby steps.

Use the most general commands possible to get the effect you want

Once you have simple steps the computer can handle, you have to figure out how to make those action work on all the text you want to modify. For instance, I could read the first line of my data and say, “There are 19 characters in the first column. Computer, go forward 19 characters to the start of the second column.” The problem is that this is a specific rather than a general command. It’s specific to the first line only. If you skip ahead to the fourth line, you can see it wouldn’t work correctly. Instead, I have to come up with a general way of getting to the position I want. This will require finding something that is true of every line of the data and using that consistency to my advantage. In this dataset, I know that every line has exactly two colons in it. I can also see that the second column always starts at the first space before the second colon. I can use this consistency to write general commands that always put me in the column space. As you can probably tell, this part of macros requires some cleverness.

Keeping these two rules in mind, here’s exactly how I think my macro should work:

1. Move to the beginning of the line
2. Set a marker at the current line
3. Jump to the second column
4. Delete from the start of the second column to the end of the line
5. Jump to the bottom of the file
6. Create a new line and paste the column data
7. Jump back up to my marker
8. Go down to the next line

These are pretty simple steps, but the real test will be converting them to Vim commands. As Vim commands, this is all pretty straight-forward except for the “jump to the second column” part, but I’ll explain it in detail below.

Here’s what the guts of my macro are going to look like, starting from visual mode (i.e., not Insert or Ex mode) on the first line of the data.

1. 0 Jump to the beginning of the line
2. mm Create a marker labelled “m”
3. f:f: Jump forward to the second colon; my cursor is now in the middle of the time string in the second column
4. bb Go back to the beginning of the word, then go back to the word before that, this puts my cursor on the number character at the end of the first column
5. f<space> Forward to the space; my cursor is now right where I want it, in between the two columns
6. d$ Delete from the cursor position to the end of the line; the text is in-memory and I can paste it somewhere else
7. G Jump to the last line of the file
8. o Create a new empty line, which puts me in Insert mode
9. <escape> Go back to Edit mode
10. p Paste the data into the line
11. 0x Jump to the beginning of the line and delete the space that I don’t need
12. 'm Jump back to the line I started the process with
13. j Go down to the next line to start the process over

Cool, I have steps that can be repeated line after line. Now it’s time to hit the docs. According to the Macros page on the Vim Tips Wiki, to record a macro, hit “q” and then a label key, just like with markers. So, I choose “s” as my label. It’s a good, memorable letter. I now type “qs”, which gives me the familiar “recording” text at the bottom left of my editor. Only this time, I actually intend for it to show up. Huzza. I’m a pro.

If I’ve designed my macro correctly, I should be able to follow it exactly, then hit the “q” key to stop recording. Of course, as a frail human, I botched the steps up the first time and had to start over. Sad panda. Life goes on. On the second try, I got the commands right. Hitting the “q” key finishes the macro.

It’s the moment of truth. I’m on the second line of my data set. The first line has already been converted. According to the macro documentation, if I want to replay my macro, I hit “@” and the macro label. Drumroll. I hit “@s” and instantly my cursor is on the third line and the second line has been converted in exactly the way I want. At the bottom of the file, the second column of the second line has been appended. It’s magic!

I know I started the process willing to do all that work myself, but now I’m spoiled. The computer is my bitch and I want it to get me some cookies and milk while I lay on the couch basking in my genius. Translation: I realize that I’m still going to have to type “@s” a bunch of times and I’m sad. There has to be a better way! And, reading the documentation further, it turns out that you can replay a macro as many times in a row as you want by declaring the number of times to repeat the “@” then the label key. So, doing some quick math, I see I have 116 more lines to convert. I type “116@qs” and BAM! The file is now one column of data. Macros are awesome.

When you think about it, if you’re doing work in Vim already, macros are pretty much just a recording of the steps you would take anyway. From now on, I’m going to think about every editing procedure I do in terms of simple, sequential actions and general commands.

The time it took me to devise the macro plus the time it took me to get it recorded was about 15 minutes, including the time it took to read the docs. Sure, that’s longer than it would’ve taken to do the conversion by hand. But, I got smarter in that time. Manual work would’ve made me more inclined to take the route of drudgery, which I’ll submit is the equivalent of getting dumber. Fifteen minutes spent learning and applying a new skill is definitely a more leveraged use of the time than reapplying a well-worn skill. If you factor in the damage done by the error I probably would’ve made doing it all manually as well as the time it would take to find and correct the error, learning to use automation is easily the cheaper way to go. The next time around it’ll be an even better option. Speaking of the next time around, now I need to create a macro to convert this one-column state I’m in into a multicolumn file using CSV format. This is going to be be fun.

header('Content-Disposition: attachment; filename='.$apk_name);
echo file_get_contents($web_inaccessible_dir.'/'.$apk_name);

Testing this code in any desktop browser, I get the expected download notification and the file downloads like a champ. When I test drive this in the Android browser the result is different. Rather than getting a file download, I get the correct filename downloading but instead of using the necessary APK extension, the file is magically transformed into an HTM file, which is completely useless. This seems like a painfully bad decision for the content handling mechanisms of the browser, perhaps even a bug. Looking for some sign in the Android logs, I see this:

D/MediaScannerService( 1386): IMediaScannerService.scanFile: /mnt/sdcard/download/that_apk_file.htm mimeType: text/html

So while this is completely bad behavior, at least there’s a trail of hope in it — MIME type. Text/HTML is not the MIME type an APK file should have and apparently the browser is making us play by the rules. I can understand that. After all, if the MIME type is the source of truth then an APK file extension is only going to make the OS confused about what to do with the file. So, the question becomes what file type should an Android APK file have?

Doing some Googling about it, I find Wikipedia has the answer. The APK file format entry explains that Android APK files should have a MIME type of application/vnd.android.package-archive. Armed with this knowledge, I tried modifying my endpoint to send the MIME type I want:

header('Content-Disposition: attachment; filename='.$apk_name);
header('Content-Type: application/vnd.android.package-archive');
echo file_get_contents($web_inaccessible_dir.'/'.$apk_name);

Hitting the endpoint now, I find that the browser gets the APK file with the APK extension. Booyah. Just for rigor, I check logcat, and see what I was hoping to see:

D/MediaScannerService( 1386): IMediaScannerService.scanFile: /mnt/sdcard/download/that_apk_file.apk mimeType: application/vnd.android.package-archive

This is good, but I notice there are still flaws in the pearl. Namely, the file downloads without an indication of filesize or progress. The downloads page shows only that the download is “in progess” all the way up to completion. On a slow connection, this will seem like the file is stuck. Not ideal.

So, knowing a little about the HTTP spec, it seems like what the browser might need is a Content-Length header to get a handle on the progress towards completion. This is a straight-forward change, requiring just a single line.

header('Content-Disposition: attachment; filename='.$apk_name);
header('Content-Type: application/vnd.android.package-archive');
header('Content-Length: '.filesize($web_inaccessible_dir.'/'.$apk_name));
echo file_get_contents($web_inaccessible_dir.'/'.$apk_name);

Giving this a whirl in the browser, I find that I now have both the correct file extension and a progress bar during the download. Huzza.

I believe I’ve got what I want, but with Android it’s not guaranteed to be that simple. Due to fragmentation, I can only assume this works on the devices I’ve tested. So far, that includes the T-Mobile G2X (2.3.3 + T-mo’s bloatware), the x86 Android Emulator running 2.2 (by the way, if you haven’t tried out the x86 Emulator for Android and are still using the ARM emulator, you need to check out http://www.android-x86.org/ – it will change your whole perspective on shit), and the HTC Sensation 4G running 2.3.4 + Sense UI. I feel like that’s a good enough start to write this post. Please comment if you have an OS version/phone that doesn’t work with this setup.

You don’t need to tip me over. You don’t need to pour me out. I’ll handle that myself. This should be posted on my site, but I like to break things up. If nothing else, it will increase my search engine visibility and make it even more likely that when someone searches for “jackson gabbard” they get nothing but me. No dead Kentuckians, no products or companies. No elementary school kids from North Carolina. Just the freelance web developer and designer.

This post comes as an interlude between work and work. Paula Dean is tasting hot peppers on TV across the room and commenting about how if he picks her nose right now, the front of her face will catch on fire. One would think her face would incur chemical burns first and probably never actually catch fire. For skin, and most things, to burn, they have to be completely desiccated. Nothing about Paula Dean seems to be that way, aside from her hair that is.

Today I encountered some word use I hadn’t heard in a long time (circumspect, assiduous, etc.). Uncommon words with highly specific meanings are treasures. As such, you can’t just go flaunting them around in mixed company. For one, doing so marks you as an elitist, a show off. Secondly, most people can’t understand you. Like Dave Chappelle once overstated in a skit, “I cain’t understaaand yoooo. Go back where you came from.” Or something like that. When you present someone with something they can’t understand that they also recognize they can’t understand because they aren’t as educated as you, well, you get a reaction.

That said, I was happy to hear that some people keep the other 90% of the English lexicon alive. The dictionary is a bit like a word museum. Once something gets pulled from public view and placed in a museum, it becomes dated and unfamiliar. Pulling the words out of their glass display cases and jamming them in there with the vocabulary of the peanut gallery once in a while does everyone some good.

It reminds the speaker of the boundaries in communication and reminds the listener that there are words beyond what the average person hears that can achieve communication more quickly. Of course, if the person doesn’t know the word then it becomes a stumbling point. Like my friend Laurie mentioned, if you have to label it (in this case, explain it), you have failed. I don’t know that I agree with that 100% though. You have failed at communicating at 100% efficiency, but you have succeeded in exposing someone to something new. There is value is broadening people’s horizons.

Speaking of altering perspectives, I’m becoming a sociopath. Actually, I suppose I should say I’m returning to sociopathy. I’m probably overstating, but the practicing sociopath in me doesn’t care how you feel about it. It’s not my concern whether you take away a perfect understanding or not. Succeeding in business requires this of me. I spent years breaking myself of the inclination to disregard the way other people feel about things I affect only to find that many of the people I work for are disregarding the way I feel. To them I say, be a wolf, get a wolf. Or, in less metaphorical terms, get a smart guy interested in not getting eaten to the point of being willing to eat you first. Sorry, compassion complicates my sociopathy, so it has to be conditionally applied.

There’s personal growth inherent in the process, of course. Controlling my emotional reaction to threats and dealing with things on a professional plane only is tricky for me. According to research, it’s tricky for everyone since we’re irrational, emotional creatures. However, I want to be a commercially successful irrational creature, so I’m having to learn to feign rationality when things heat up so I don’t do something impulsive that costs me in the long run. When someone tries to take advantage of me, I have to see the opportunity in the adversity (thanks for the reminder Einstein) and make the situation work for me. If I can’t make it work at all, I have to walk away.

The compassionate person in me wants to help the advantage-taker see what they are doing, the victim in me wants to defend myself Malcolm X style, and the eager beaver in me wants to make the person happy, even if that means giving everything away. The business man in me has to win and find the solution that resolves the situation as much as possible without sacrificing me, my time, or any money. Then, after I get the check, I can revisit thoughts of any means necessary and education.

This is very unlike me. If you’re doing something awful, I want to help you see how and why. I don’t want to just make you behave how I want you to. However, I’m dealing now with adults whom it’s not my responsibility to raise. Or at least, whom it’s not my business’ responsibility to raise. So, the wolf gets to call the shots. I suppose a better metaphor is the fox though, they’re much cleverer. Maybe some sort of hybrid–the brains of the fox, the body and bite of the wolf. Wolfox? Folf? Wolx? I suppose a clever conjoining of the nouns isn’t requisite for the idea to sink in, but it would be funner.

Whether it takes a wolf, a fox, or a manatee, I have to keep my eyes open, my pulse level, and my sociopath switch ready to flip. You might want something, but if you didn’t pay for it, you’re not getting it. Even if that means I have to fire you. Lesson learned there.

I realize now that I’ve just made a faux pas (fox paws?). ‘Whether’ followed by three options? What sort of self-respecting English degree holder am I? Every Tom, Dick, and Harry knows the etymology of ‘whether’ hearkens back to the phrase ‘which of two,’ much like ‘either’ and ‘neither.’ ‘Whether’ followed by three options. What’s next–dilemma with four alternatives? Sheesh. Please accept my deepest apologies for the error. Though, if you desire some sort of recompense jingling in hand for the mistake, you might instead find a brief and brisk application of the canines.