gaedocstore: JSON Document Database Layer for ndb

In my professional life I’m working on a server side appengine based system whose next iteration needs to be really good at dealing with schema-less data; JSON objects, in practical terms. To that end I’ve thrown together a simple document database layer to sit on top of appengine’s ndb, in python.

Here’s the github repo: https://github.com/emlynoregan/gaedocstore

And here’s the doco as it currently exists in the repo, it should explain what I’m up to.

This library will no doubt change as begins to be used in earnest.

gaedocstore

gaedocstore is MIT licensed http://opensource.org/licenses/MIT

gaedocstore is a lightweight document database implementation that sits on top of ndb in google appengine.

Introduction

If you are using appengine for your platform, but you need to store arbitrary (data defined) entities, rather than pre-defined schema based entities, then gaedocstore can help.

gaedocstore takes arbitrary JSON object structures, and stores them to a single ndb datastore object called GDSDocument.

In ndb, JSON can simply be stored in a JSON property. Unfortunately that is a blob, and so unindexed. This library stores the bulk of the document in first class expando properties, which are indexed, and only resorts to JSON blobs where it can’t be helped (and where you are unlikely to want to search anyway).

gaedocstore also provides a method for denormalised linking of objects; that is, inserting one document into another based on a reference key, and keeping the inserted, denormalised copy up to date as the source document changes. Amongst other uses, this allows you to provide performant REST apis in which objects are decorated with related information, without the penalty of secondary lookups.

Simple Put

When JSON is stored to the document store, it is converted to a GDSDocument object (an Expando model subclass) as follows:

  • Say we are storing an object called Input.

  • Input must be a dictionary.

  • Input must include a key at minimum. If no key is provided, the put is rejected.

    • If the key already exists for a GDSDocument, then that object is updated using the new JSON.
    • With an update, you can indicate “Replace” or “Update” (default is Replace). Replace entirely replaces the existing entity. “Update” merges the entity with the existing stored entity, preferentially including information from the new JSON.
    • If the key doesn’t already exist, then a new GDSDocument is created for that key.
  • The top level dict is mapped to the GDSDocument (which is an expando).

  • The GDSDocument property structure is built recursively to match the JSON object structure.

    • Simple values become simple property values
    • Arrays of simple values become a repeated GenericProperty. ie: you can search on the contents.
    • Arrays which include dicts or arrays become JSON in a GDSJson object, which just hold “json”, a JsonProperty (nothing inside is indexed, or searchable)
    • Dictionaries become another GDSDocument
    • So nested dictionary fields are fully indexed and searchable, including where their values are lists of simple types, but anything inside a complex array is not.

eg:

ldictPerson = {
    "key": "897654",
    "type": "Person",
    "name": "Fred",
    "address": 
    {
        "addr1": "1 thing st",
        "city": "stuffville",
        "zipcode": 54321,
        "tags": ['some', 'tags']
    }
}

lperson = GDSDocument.ConstructFromDict(ldictPerson)
lperson.put()    

This will create a new person. If a GDSDocument with key “897654” already existed then this will overwrite it. If you’d like to instead merge over the top of an existing GDSDocument, you can use aReplace = False, eg:

    lperson = GDSDocument.ConstructFromDict(lperson, aReplace = False)

Simple Get

All GDSDocument objects have a top level key. Normal ndb.get is used to get objects by their key.

Querying

Normal ndb querying can be used on the GDSDocument entities. It is recommended that different types of data (eg Person, Address) are denoted using a top level attribute “type”. This is only a recommended convention however, and is in no way required.

You can query on properties in the GDSDocument, ie: properties from the original JSON.

Querying based on properties in nested dictionaries is fully supported.

eg: Say I store the following JSON:

{
    "key": "897654",
    "type": "Person",
    "name": "Fred",
    "address": 
    {
        "key": "1234567",
        "type": "Address",
        "addr1": "1 thing st",
        "city": "stuffville",
        "zipcode": 54321
    }
}

A query that would return potentially multiple objects including this one is:

GDSDocument.gql("WHERE address.zipcode = 54321").fetch()

or

s = GenericProperty()
s._name = 'address.zipcode'
GDSDocument.query(s == 54321).fetch()

Note that if you are querying on properties below the top level, you cannot do the more standard

GDSDocument.query(GenericProperty('address.zipcode') == 54321).fetch()  # fails

due to a limitation of ndb

If you need to get the json back from a GDSDocument, just do this:

json = lgdsDocument.to_dict()

Denormalized Object Linking

You can directly support denormalized object linking.

Say you have two entities, an Address:

{
    "key": "1234567",
    "type": "Address",
    "addr1": "1 thing st",
    "city": "stuffville",
    "zipcode": 54321
}

and a Person:

{
    "key": "897654",
    "type": "Person",
    "name": "Fred"
    "address": // put the address with key "1234567" here
}

You’d like to store the Person so the correct linked address is there; not just the key, but the values (type, addr1, city, zipcode).

If you store the Person as:

{
    "key": "897654",
    "type": "Person",
    "name": "Fred",
    "address": {"key": "1234567"}
}

then this will automatically be expanded to

{
    "key": "897654",
    "type": "Person",
    "name": "Fred",
    "address": 
    {
        "key": "1234567",
        "type": "Address",
        "addr1": "1 thing st",
        "city": "stuffville",
        "zipcode": 54321
    }
}

Furthermore, gaedocstore will update these values if you change address. So if address changes to:

{
    "key": "1234567",
    "type": "Address",
    "addr1": "2 thing st",
    "city": "somewheretown",
    "zipcode": 12345
}

then the person will automatically update to

{
    "key": "897654",
    "type": "Person",
    "name": "Fred",
    "address": 
    {
        "key": "1234567",
        "addr1": "2 thing st",
        "city": "somewheretown",
        "zipcode": 12345
    }
}

Denormalized Object Linking also supports pybOTL transform templates. gaedocstore can take a list of “name”, “transform” pairs. When a key appears like

{
    ...
    "something": { key: XXX },
    ...
}

then gaedocstore loads the key referenced. If found, it looks in its list of transform names. If it finds one, it applies that transform to the loaded object, and puts the output into the stored GDSDocument. If no transform was found, then the entire object is put into the stored GDSDocument as described above.

eg:

Say we have the transform “address” as follows:

ltransform = {
    "fulladdr": "{{.addr1}}, {{.city}} {{.zipcode}}"
}

You can store this transform against the name “address” for gaedocstore to find as follows:

GDSDocument.StorebOTLTransform("address", ltransform)

Then when Person above is stored, it’ll have its address placed inline as follows:

{
    "key": "897654",
    "type": "Person",
    "name": "Fred",
    "address": 
    {
        "key": "1234567",
        "fulladdr": "2 thing st, somewheretown 12345"
    }
}

An analogous process happens to embedded addresses whenever the Address object is updated.

You can lookup the bOTL Transform with:

ltransform = GDSDocument.GetbOTLTransform("address")

and delete it with

GDSDocument.DeletebOTLTransform("address")

Desired feature (not yet implemented): If the template itself is updated, then all objects affected by that template are also updated.

Deletion

If an object is deleted, then all denormalized links will be updated with a special key “link_missing”: True. For example, say we delete address “1234567” . Then Person will become:

{
    "key": "897654",
    "type": "Person",
    "name": "Fred",
    "address": 
    {
        "key": "1234567",
        "link_missing": True
    }
}

And if the object is recreated in the future, then that linked data will be reinstated as expected.

Similarly, if an object is saved with a link, but the linked object can’t be found, “link_missing”: True will be included as above.

updating denormalized linked data back to parents

The current version does not support this, but in a future version we may support the ability to change the denormalized information, and have it flow back to the original object. eg: you could change addr1 in address inside person, and it would fix the source address. Note this wont work when transforms are being used (you would need inverse transforms).

storing deltas

I’ve had a feature request from a friend, to have a mode that stores a version history of all changes to objects. I think it’s a great idea. I’d like a strongly parsimonious feel for the library as a whole: it should just feel like “ndb with benefits”).

 

Advertisements
gaedocstore: JSON Document Database Layer for ndb

Moore Solar!

The Sun!

The outlook for the world in terms of energy security and global warming seems pretty bleak. But I think there’s a light on the horizon, and that light is the sun. Amongst all the renewable energy areas, solar seems to me to be the only one with exponential characteristics (cost reductions, adoption), and the only one that can tap the consumer capital (trillions/year in the US alone).

I thought I’d have a go at quantifying the picture. This is fairly BOTE stuff, but still interesting. Note also that I’m not saying Solar is definitely what we are going to do, but that if we do nothing else, we will do solar.

The technium wants more power, a lot more power. Fossil is not going to do the job from here on out. So a renewable buildout will happen, and it’ll blow what we have now out of the water. Our future is not energy poor.

Anyway, enough of the rhetoric, here are the calcs:

——

From reading/watching Saul Griffiths, things can seem gloomy. See his slides here.

He says, amongst other things, that for us to stabilize at 2 degrees C temperature rise, we need to move entirely to renewables over the next 25 years.

We use 18 TW of power on the earth at the moment. He’d like us to actually reduce our power usage, but I’d say we’d have to plan to do about double that, at least.

18TW - breakdown of use

Total energy available

In energy terms, 18TW is about 155,880TWh / year. So we need to hit a target of roughly 300,000TWh / year globally from renewables (and no fossil) by about 2037. That’s about 34TW.

There’s a lot of talk about solar installation going exponential. Gregor McDonald shows a graph that looks exponential (he says parabolic, not sure why) showing an improvement rate conservatively of 1.6 increase / year in total global solar installation, and some specific figures; eg: in 2011, the total world solar output was 55.7 TWh. Small.

Global Solar Consumption in TWh 2001-2011

But if we extrapolate using an increase of 1.6/year (factor of 10 every 5 years), we get this:

Year Global Solar TWh / year
2010 29.90
2015 365.04
2020 3,827.67
2025 40,136.08
2030 420,857.30
2035 4,413,008.65
2040 46,273,749.60
2045 485,215,432.64
2050 5,087,852,574.96

Year Global Solar TW
2010 0.00 – so small it rounded to zero)
2015 0.04 – looking grim
2020 0.44 – still pretty grim
2025 4.58 – more than current world electrical power of 2 TW
2030 48.04 – more than the 34 TW we need before 2037
2035 503.77 – a stupendous renewable power output, a golden age on earth
2040 5,282.39 – impossible to understand the implications of this, but maybe also we can’t get here.

85,000 TW (see Saul’s slides above) is the cap of all the solar that hits the earth. Something less than this is what we can harvest. Let’s say we can cover 10% of the earth with 10% efficient solar, that’s 850TW. I expect the efficiency will get higher; we can’t expect that what we build out in the 2030s will be as poor as what we use now. So maybe we can go further? Even if we can’t, these are very, very large numbers! Probably enough to get us off planet easily, collecting power in space (the earth doesn’t actually trap very much of the output of the sun!).

So basically this progression will look tragically awful until the early 2020s, when it’ll start looking promising, then we switch entirely to solar, then shortly after that energy becomes totally abundant; what on earth would we do with it all???

Moore Solar!

The Ballad of John Henry

When John Henry was a little baby boy, 
        sitting on the his papa's knee
Well he picked up a hammer and little piece of steel
Said Hammer's gonna be the death of me, lord, lord
Hammer's gonna be the death of me

This is a post about the destruction and recreation of the world.

I’ve been building a diddly bow. It’s a one-stringed instrument originally invented by african slaves in America’s old south, and the cool thing about it is that you can put it together out of bits of stuff you find around the place; a big plank of wood, nails, bottles, smaller chunks of wood. And some wire, ideally music wire of some kind, really ideally a guitar string.

Now I want to make some more of them, for which everything is easy except the string. I’m using a guitar G-string; I thought today I’d head off to a music shop and grab some more guitar strings, and maybe one for a bass guitar?

So anyway, I went into the city (Adelaide, South Australia), and schlepped into Allens Music, in Gawler Place, right in the CBD. Anything from Allens is going to cost an arm and a leg, but hey, they’ll have some strings, and how much can they be? Note that I have never bought guitar strings, but they seem to be about $10 for a set online, give or take, so how bad can it be?

Actually I hardly ever buy any musical gear from a shop. I buy everything online. I would have bought the strings online, but I needed to get this project moving.

The captain said to John Henry
I'm gonna bring that steam drill around
I'm gonna bring that steam drill out on the job
I'm gonna whup that steel on down

When I arrived, the place was a shambles. Down mouthed, black shirted sales people, scrubbing stuff off the windows, packing stuff up. It’s a three story shop (ground floor guitars, basement synths and mics, upstairs pianos, sheet music, classical instruments). Today, upstairs and basement gone, shut down, and ground floor was like a garage sale.

“No guitar stuff left” says the guy at the till. I asked them a bit about it. No, they’re not moving. Apparently Allans Billy Hyde has completely collapsed, is being entirely shut down.

From this link:

The receivers of collapsed music chain Allans Billy Hyde have announced all the company’s stores will close and more than 500 jobs will be lost as a result.

Brendan Richards of Ferrier Hodgson said in a statement Australian Music Group Holdings will be shut down.

“The loss of jobs is disappointing, but we exhausted all avenues and there is no other way forward for this business,” he said.

“These people have served music lovers and been a key part of the Australian music industry for generations. It is a sad day for live music in this country.”

The 513 staff will be made redundant over the next few weeks. All the 27 stores will be closed.

The company collapsed earlier this year, after an injection of capital failed to save the business and investors called in receivers. It had been experiencing trouble for years, with the two groups, Allans and Billy Hyde, creating a new chain in 2010.

So I guess everyone there was on the last day of their job.

John Henry told his captain
Lord a man ain't nothing but a man
But before I'd let your steam drill beat me down
I'd die with a hammer in my hand

John Henry is a great song that I’ve just discovered recently. The story is a classic American style; a heroic big man, greater than ordinary mortals. Wikipedia says:

John Henry is an American folk hero and tall tale. Henry worked as a “steel-driver”—a man tasked with hammering and chiseling rock in the construction of tunnels for railroad tracks. In the legend, John Henry’s prowess as a steel-driver was measured in a race against a steam powered hammer, which he won only to die in victory with his hammer in his hand.

He’s got a touch of Paul Bunyan to him, but he’s a tragic figure. Mightier than the steam drill, he succumbs to his exertions, while as we know the steam drill digs on.

John Henry asid to his shaker
Shaker why don't you sing
Because I'm swinging thirty pounds from my hips on down
Just listen to that cold steel ring

There’s an obscure name for the computer that is sometimes used by computer scientists: the General Purpose Machine. It’s called that because this machine, unlike all others before it, can be programmed to do anything that any special purpose machine you might conceive of could do. So it solves an entire class of problems in one shot (as long as we hand-wave around the programming). And that entire class of problems turns out to be anything we can automate. And that in turn might turn out to be Everything.

Oddly enough the world hasn’t actually figured this out, and you can forgive it for that. Progress is slow (in day to day terms) and uneven, because, after all, we can’t hand-wave around the programming, and Moore’s Law is fast, but it’s doesn’t seem that fast when you’re slogging through your day-to-day life. Some days you mightn’t even think about computers at all!

Now the captain said to John Henry
I believe that mountain's caving in
John Henry said right back to the captain
Ain't nothing but my hammer sucking wind

And we also seem to have a bias toward believing that our institutions are solid. They always have been before, right?

And then, the newspapers stumble. Encyclopedias, well is that still a business?

Let’s not even mention the music publishing industry! Oh, ok, let’s.

There are a lot of theories about what’s brought the music publishing industry undone. Piracy! Apparently limewire alone owes the music industry 75 Trillion Dollars. That surely is a lot of money.

Or how about iTunes, killing the music industry with kindness? Jon Bon Jovi thinks Apple killed music by making it less like when everything totally sucked. But some people who aren’t total idiots have made a better argument that the revenue from digital is just nothing like that from physical media:

I have another idea about this. My own experience is that suddenly, especially since about 2005 when Youtube started (yep, only 7 years ago), I have access to more or less the entire back catalogue of recorded music history. For free. So now instead of listening to the radio, and/or buying the “latest” stuff, I’m listening to this cornucopia of music from the last hundred years. And I see everyone I know doing the same thing. So I hear of a band or a musician that I like, then follow back through their influences, to those people’s influences, and etc until I’m listening to blues from the thirties, or early vocalese, or watching Mahalia Jackson on the Nat King Cole show.

And it’s clear from this vantage that the music industry’s modus operandus has always been to promote the new before the good. Keep taking the old stuff off the shelves, replace it with the new stuff. Keep hammering you with this month’s prodigy, while you whip last month’s out the back door.

Only we don’t have to put up with that any more.

Without this disruption I’d never have heard of John Henry.

Now the man that invented the steam drill
He thought he was mighty fine
But John Henry drove fifteen feet
The steam drill only made nine

What I do is, I make steam drills.

In the tech industry, we’re always disrupting someone. Someone’s job is going, someone’s industry is being replaced/canibalised. It’s a ruthless, heartless kind of thing.

We don’t notice because we disrupt ourselves most of all. My own career path is nothing like anyone’s I’ve seen from other industries. I think of myself as a professional forgetter. I have to pick up new technologies at a moment’s notice, pick up new languages, pick up entirely new paradigms of computing (eg: Platform as a Service).

And then drop them. These skills used to last 5 to 10 years. Now they last, what, 3? 2? The normal technical skill useful lifespan is shortening, towards the minimum timespan required for mastery.

It’s an industry littered with those who couldn’t keep up. Hordes of one-language techies, one trick ponies, stuck in legacy land or eventually on the scrap heap. Legions of used-to-be-programmer managers, with slowly aging skills, also staring irrelevance and unemployability direct in its dead eye. Mountains of legacy code, built for a time long ago (eg: 2007).

Eventually of course we’ll all put ourselves out of work for good. But when we do that, everyone else will come with us. Intelligence will be something you download on your whatever-phones-are-by-then.

John Henry hammered in the mountains
His hammer was striking fire
But he worked so hard, it broke his poor heart
And he laid down his hammer and he died

Here’s the smartcompany article about Allans Billy Hyde from waaaay back in August:

Another retailer has bitten the dust. Australian Music Group Holdings, trading as the iconic Allans Billy Hyde brand, has been placed in receivership.

Rumours of the company’s demise were floating around in March, but joint managing director Tim Mason told SmartCompany at that time the company had received an injection of capital, debt was reduced, and the business was trading fine albeit in rough conditions.

The fact a business of this size and calibre – it owns 25% of the market – can be struck by the retail downturn demonstrates the strength of pessimistic consumer confidence but also the ramifications of offshore retail.

Ferrier Hodgson confirmed yesterday the company had been placed in receivership – and those same harsh retail conditions are to blame.

“Things are pretty rough in retail right now,” Ferrier Hodgson partner James Stewart told SmartCompany.

“The business had been recapitalised as far back as March, and the business was not travelling at levels the people in charge would have liked it to travel at. The stakeholders decided to call it a day.”

“Our intention is to seek a buyer for the business as soon as we possibly can.”

The odd thing is, I never noticed until now, and that only by accident. Or perhaps it’s the opposite of odd?

Now John Henry had a little woman
Her name was polly Anne
John Henry took sick and had to go to bed
Polly Anne drove steel like a man

I took my darling wife back to Allans to pick over the almost bare carcass. She said to me guiltily that she feels a little responsible. She buys a lot of sheet music for singing students and choirs, but it’s all from SheetMusicPlus. Or else free from the choral public domain library and similar. In the end, it’s cheaper and more convenient.

Of course this idea of buying sheet music makes no sense either. SheetMusicPlus fills a niche, but it wont last, like everything else. Something could happen sooner, but eventually pieces of paper will be replaced in choirs and orchestras by the stupidly cheap eink based A4 sized descendants of today’s tablets. They might stick around selling DRM infested skeuomorphic bundles of faux paper to tablet owners, but how long can such an artificial situation really continue? Depressingly, longer than is warranted, but it’s by no means permanent.

***

We’ve seen the slide of the music industry, encyclopedias, newspapers.

Now we’re watching cars, movies, retail, univerities, all stumbling.

Look forward to this for formal schooling, world finance, and governments.

John Henry had a little baby
You could hold him in the palm of your hand
And the last words I heard that poor boy say 
My daddy was a steel driving man

So the cool thing was, that from the husk that once was Allans Music, I bought some strings for an orchestral bass. Apparently they normally cost $270 at Allans, for a set of 4!, but you can get them from Amazon for $115. Probably there’s a better price if you shop around a bit.

It seems like sacrilege to use these strings on a didley, but they’ll make a great sound, and Allans sold them to me for $30. Because they’ve gone broke. Because the world is quietly getting on with the upheaval of everything, even stuff you like, even stuff you think the world is entirely built on.

So every Monday morning
When the blue birds begin to sing
You can hear John Henry a mile or more
You can hear John Henry's hammer ring
The Ballad of John Henry

Ermahgerd Jerver!

This is the second in a series about my hilarious hijinx crossing over from hairy cave dwelling Ubuntu user to latte drinking Windows 8 man-of-win. See the original post here: Windows 8 Madness

So, first thing I’ll really need tomorrow morning is a dev environment. That means the unlovable Eclipse, and then various plugins. And git. And I’ll need to do something about some bash scripts. But first, Eclipse.

This is easy, it’ll be just like ubuntu. Just open up the store, type “Eclipse”, press “Go” or whatever.

Oh.

So how’d I do this in the old days?

Ok, I head over to eclipse.org, download eclipse (like 47 steps later…), just classic eclipse, no bundled stuff.

It comes down as a zip. Extracting that in Win 8 was sort of weird. Maybe I’m missing something.

Then I copy it to a folder somewhere (I use c:\programs, have done since c:\program files got weird). Run eclipse.exe, and then…

Past wars reverberate even now…

Oh sweet baby jeebers. There’s no Java. I have to remember how to do this.

That’s ok, it’ll be in the store…

Google “download java”. End up on Oracle’s website, the Java bit, where Oracle wears Sun’s skin like a suit (It’s very slimming on you Larry).

Which is… what… urm…

sudo apt-forget-this-shit -wtf

Anyway, I downloaded it, ready to install, but that’s enough of this for now. Tomorrow’s coding will have to be with Ubuntu, I’ll try this again tomorrow night.

Meanwhile, it’s time to watch a show with my darling wife. She wants to watch something on ABC iView. Alright, this is easier.

So hey, it’s on the website, and there’s some kind of flash player (actually it’s Adobe’s apparently and I’m not totally loving that). We’ll watch it on that.

Let’s hope tomorrow is better.

 

Ermahgerd Jerver!

Windows 8 Madness

Ok, I’ve finally lost it.

I’m typing this on Windows 8. I decided to give it a shot. I’m probably going to regret this.

Now, I’m not a n00b to Win 8. I’ve had Windows 8 Release Preview installed for the last few months, on my dual boot windows machine. My main OS is Ubuntu, 12.04, but I have windows on here for the classic reason… to play games.

Not that I play many, mind you. But I do want to be able to run Starcraft II and Diablo III.

Anyway, I’ve played with it a bit. My first impression was “Shiny! Love the Store! Could get used to the tiled thing”. I used it to watch SBS (an Australian television station whose transmitters are all broken or underground), that was kind of cool.

But once I got past the initial fun stage, I noticed that it was kind of annoying. Too little configurability. The apps are weird. And having everything fullscreen all the time on a big monitor just sucks so much I can’t really cover it. It sucks more than Facebook. It’s a bad idea. In fact here’s me using WordPress to write this article right now, just as a picture of how bad it sucks:

See how my edit window is massively wide? That’s because I’m stuck on full screen. WordPress resizes the middle column (including the text editing)  to fill the screen. And I can’t fix this.

Well, I can, I could use Chrome in Desktop mode rather than Metro, but I’m trying not to do that. I want to give Metro a proper go; that means not using this thing like Windows 7’s retarded child, but rather using Metro as often as possible.

Hrm, I’ve wandered off course…

Anyway, I just got my hands on the full, released Win 8. I upgraded my release preview, and thought hey, why not give this thing a red hot go? That’d be fun!

Prehistory

I started using windows in the 80s. My dad had an IBM compatible pc from work, which ran Window 3.0 iirc (might have actually been win 2.0!). Before 3.1 and 3.11, anyway; before it was cool.

Figure 1

EDIT: Someone pulled me up about Windows never being cool. I beg to differ, see Figure 1!

Anyway, I bought my first Windows PC when I first moved out of home. It was a 486sx 25mhz I think, cost me $4000 including a printer (and I’ve never had the money to buy such an expensive machine since!). Man, I loved that machine. Wolfenstein. Doom. Need I say more? (well, ok, Turbo Pascal.  Borland C++ 4.0 on 30+ floppy disks. A good modem).

Not long after that I started working as a windows developer. Access, then Delphi and C++ and VB (bleh), and then etc. Windows programming fed my young family through the 90s and noughties.

The microsoft programming world is a bit culty and a bit cut off from everything else – when you’re in it, everything is MS. Which programming language will I use, C# or VB.Net? Which database will I use, SQL Server or … well, SQL Server? And etc.

But in the noughties my spidey sense began to tingle. The outside world, particularly the unices, and the world of web dev, were bleeding in. Look, there are choices!

I was an early adopter of cloud computing; I jumped into gmail in 2004, and had as much of my stuff online as possible. I used to sync my office docs up to some online virtual server space, and I began using online source control for all my code pretty early (remember sourceforge?)

But I was scared of Linux. I’d used unix at uni, and even had a job with SCO Unix (boo hiss!) in the early 90s. But I was ignorant of Linux. It struck me that I needed to pull it together and check this stuff out. And the only real way to do that is to use it on a daily basis.

I was ready to try it, and I even had a way to cope with my daily need for a windows programming environment; VMWare Desktop. Totally magical software (especially at the time), I was already doing all my coding in VMs anyway (makes it easier to throw out a crufty dev environment and rebuild from scratch, or switch host machines where necessary).

And then came the push I needed.

Windows stopped being fun

My new Dev notebook came with Vista. Which as a .net developer I was hanging out for. The .net framework actually shipped with Vista! It was no Longhorn, but it was a shiny new version of my favourite OS, the next NT. More secure, more .net friendly, we were truly headed to the promised land!

Yeah, and then reality. It sucked to use on a daily basis. Slooooow. Bad.

And clearly, the OS was no more .net friendly than XP had been. Just more Win32 shitola in a shiny, laggy wrapper. Really????

It was time to jump ship.

Ubuntu!!!

Ubuntu was a revelation. I jumped in at Intrepid Ibex. I really knew nothing about it, but I downloaded and burned a live CD, ran it (worked!), then installed the real thing. It went in alongside windows without a hitch, it was pretty easy, it just detected my hardware and worked.

Now it was nothing glorious graphics-wise, but I slowly learned that, with Windows running in VMWare, it was perfectly serviceable (in fact windows ran better in a vm with a linux host). Not only that, but I didn’t have to worry about being pwned (really! it took me a while to believe it), and after a while I figured out that you don’t download and install software except as the last resort, instead use the debian package manager (and synaptics). Wow. Wow wow wow.

Burning a CD by just installing free excellent software from the standard repos; a revelation. All the utilities just there at my fingertips. Amazing. For a technically switched on person, a debian based distro is just massively empowering.

I’ve used Ubuntu as my main OS since then, on a succession of beasty dell laptops, up to my current setup featuring Ubuntu 12.04 (I’m conservative with my desktop, I’m waiting until they push 12.10 out through auto updates) and Win 8.

And do you know, once I used Ubuntu all the time, I wanted to stop using windows. I tried Monodevelop and Sharpdevelop for linux friendly “.net” coding (really mono development), but really for proper .net it’s Visual Studio or bust. Which is windows only.

Instead I started building webby stuff on Google’s AppEngine (which is *awesome*), and then eventually I snagged a job doing LAMP stack stuff (php, urgh), and then AppEngine and Android (golden!). I’m still there, it’s brilliant!

Dr Phil Lunacy

But something serious has changed in Windows land, and that’s Windows 8. From all I’ve read of it, it’s quite an upsetting release for a .net developer (I’m assuming the .net part of the company are now Stephen Sinofski’s sex slaves in a dungeon somewhere, forced to listen to Songsmith tunes all day). But there’s a make-or-break about this thing, and all sorts of noises out there about it, so I feel like I need to know something about it.

And that means living with it for a bit.

I don’t look all that much like Ballmer, do I?

Dr Phil is usually a man of few words, but he does have one piece of homespun advice, and it’s this:

Love every idea for 15 minutes.

Now that idea for me, right now, is Windows 8, and I’ll give it a few weeks (because I’m a good guy).

So what I’m going to do is to try using this thing as my main environment. It needs to fill my media consumption / entertainment needs, my officey needs, and it needs to serve as my dev environment.

For entertainment, I watch online streaming shows with my darling wife, as well as some downloaded shows. Basic media stuff. Plus, my games should work, but that’s not super critical (getting a bit bored of them atm).

For officey stuff, it needs to do all things google. Gmail, gdrive, google apps for business. Google+ ! I use these exclusively (I don’t even use Open Office any more), but that should be easy, Chrome oughta do the job.

Lastly, development, and that’s the toughy. I stopped using a vm for development since I moved to doing non-windows stuff. It turns out that linux just gets less crufty that way. And moving back to Windows, I want to see Win 8 step up to the challenge of my daily development routine. So it needs to cope with Eclipse. I do Python development on AppEngine at the moment, which means it needs to cope with that (python + pydev + google plugin for eclipse) and I also do Android dev (so Java plus ADK). Oh, and I use ssh quite a bit to talk to remote machines. And some custom bash scripts (I’m prepared to rewrite those into something windowsy, hopefully not batch files!). And Git!

It occurs to me that I don’t even know how to find a command prompt in this environment.

Anyway, I’ll write a diary of how this goes, on this blog.

Next: Ermahgerd Jerver!

Windows 8 Madness

Adelaideans, please help! And get a free CD.

I need help!

My darling wife Jodie O’Regan and I have a most excellent concert approaching.

It’s on the Monday public holiday (1 October), at the Church of the Epiphany in Crafers (map link), at 7pm. And it’s Free!

We’re singing with the Queensland Kodaly Choir, who I just saw tonight at St Johns, Halifax St in the city. They’re awesome, beautiful, something you really want to see.

And of course Jodie and I will be doing a bunch of our crazy two-voice stuff (see previous post for examples). We’ll also be singing some three-voice a capella material with our good mate Chris Bradley who’s come to Adelaide with the Kodaly folk.

Anyway, I still need help!

The problem is, we need an audience, and it doesn’t look like we’re getting a huge turnout. It’d be a terrible pity if this concert (which I quietly think will be a properly magical experience) didn’t have a crowd, particularly for the Kodaly choir, our guests.

So if you can possibly manage it, do come along on Monday night. If you can’t, then tell someone local who might be able to make it.

Just to sweeten the deal a little, I’ll have some demo CDs there, which you can just take home. They might be a little undecorated (I’m running them off on my laptop), they’re our work in progress, but they’re red hot, the real deal. You totally want one, honestly!

So spread the world, and come along. We’d love to see you, it’ll be a spectacular night, you’ll pity the poor folks who missed out!

Adelaideans, please help! And get a free CD.