About Tidey

I’d like to introduce Tidey.

Tidey is a tide prediction application that uses xtide to make tide predictions and provides a fun web UI.

Tidey is a 3-fold personal experiment. The goals I had/have for Tidey are:

• Build a project using pure Tornado (i.e. not mixed with Django) to evaluate it
• Practice using RaphaelJS and JSON
• See what kind of advertising revenue a simple site can generate with a small amount of work

Tornado is cool, but only for certain applications. I wouldn’t want to build an application like a church management system using Tornado. Django provides the absolutely amazing built-in admin tool that is indispensable IMHO. Tornado has no such thing. That said, Tornado is useful for certain types of applications and used in conjunction with Django could prove very useful. For this application I am using it to asynchronously run xtide in the background. The server itself is a single thread process.

Raphael JS is very cool.

Only time will tell on the advertising.

I spent about 20-30 hours on this project. I would still consider it beta.

Asynchronous Shell Commands with Tornado

I’ve been playing around with Tornado a bit and wanted to asynchronously call a long-running shell command without blocking the server process. Here is my solution.

Running this server I am able to visit http://localhost:8888/test/ numerous times while a request for http://localhost:8888/ is waiting for the command to finish. The beauty of this is that this is a single process with a single thread. With a process this light and fast, a relatively large number of applications can be crammed into a $10/mo hosted account.

#!/usr/bin/env python
import os
import tornado.httpserver
import tornado.ioloop
import tornado.web

class MainHandler(tornado.web.RequestHandler):
    @tornado.web.asynchronous
    def get(self):
        self.ioloop = tornado.ioloop.IOLoop.instance()
        self.pipe = p = os.popen('sleep 5; cat /etc/mime.types')
        self.ioloop.add_handler( p.fileno(), self.async_callback(self.on_response), self.ioloop.READ )

    def on_response(self,fd,events):
        for line in self.pipe:
            self.write( line )

        self.ioloop.remove_handler(fd)
        self.finish()

class TestHandler(tornado.web.RequestHandler):
    def get(self):
        self.write('this is a test')

application = tornado.web.Application([
    (r"/", MainHandler),
    (r"/test/", TestHandler),
])

if __name__ == "__main__":
    http_server = tornado.httpserver.HTTPServer(application)
    http_server.listen(8888)
    tornado.ioloop.IOLoop.instance().start()

We’re off to see the Scientist

If you haven’t heard about the ClimateGate scandal, then you’ve been hiding under a rock.

Late last week hackers penetrated systems at the Climate Research Unit (CRU) at the University of East Anglia in the UK. The CRU is considered something of a global hub for climate-change “science.” What the released email revealed is that climate scientists have been deliberately deceptive and scientifically disingenuous in their campaign to stop “climate change.” Contrary data has been concealed or even tampered with to hide evidence contrary to climate change theories.

Science in the strictest sense is the objective application of the scientific method (i.e. observation, hypothesis, testing, lather-rinse-repeat). A scientist is someone who is understood to objectively follow the scientific method. But for the average lay person science is “whatever scientists say.” The average lay person doesn’t understand the advanced mathematics behind the climate models and thus can never really understand the “science.” He has to trust the scientist to tell him what he knows. That is “science.”

In today’s world “science” is held in very high regard. Few dare to question the teachings of “science.” “Science” has produced space travel, the internet, modern health-care, and other wonders too numerous to mention. But what people haven’t realized en masse is that “science” is really a matter of faith… in scientists.

Even though we can see the evidence of many scientific successes, entire fields of “science” do not produce tangible results. The present case of climate change “science” in particular is a purely speculative theory with no possibility of proof (and consequently it does not follow the scientific method). Similar examples are string theory, Darwinian evolution, and big bang theory.

While on the one hand we are given many tangible wonders like iPhones and cars to dazzle the mind and lend “science” credibility, on the other hand we are fed unproven theories as factual doctrine.  While the wizards of science put on a wondrous show for us with all the whiz-bang gadgets, when we pull back the curtain we see only a fraud.

There is no wizard.

There is no scientist.

Sheepology Structure

Things have changed a bit since I last created a structure diagram for Sheepology. This graph was generated by django-command-extensions and Graphviz.

Sheepology Version Control

Version control seems to be broken in the demo right now. It’s probably due to all the version changes of Django and such that have happened since last I test this feature.

UPDATE: Versioning is fixed. I had to install django-reversion 1.2 so it would work with Django 1.1.1.

Sheepology Updates

I just rolled out an update to the Sheepology demo. Many changes have taken place since this demo was last updated:

• Content management system – I needed this ASAP so it was what I’ve worked on for the last month or so.  It doesn’t have a lot of features right now, but has an architecture that I am very happy with and is highly extensible.
• Revamped the communications architecture.
• Generalized the Flex-based visual tree editor so that it could be used with more than just groups. It is now being used for content categories as well. (I would actually like to rewrite this in Javascript now that I have played with raphael.js)
• Added version control for pretty much anything of importance (people, groups, events, content, etc.) using django-reversion. You can easily restore deleted people, content, whatever, or revert to previous versions.
• Updated the theme to the latest version of Grappelli
• Optimized some queries
• Got the whole thing running under Tornado instead of Apache. It now uses much less memory and runs in a single process. In theory it should be very fast but I haven’t load tested it yet.

I hope to role this out into production in my church (the “public” site in the demo is a previews of the site I’m doing for my church).

The list of features I want to add is growing faster than the list of items completed, but I think I need to get something into production and get it field tested before going too much further.

More on The Kindle

I take back everything nice I said about the Kindle.  Don’t buy one.

This morning, hundreds of Amazon Kindle owners awoke to discover that books by a certain famous author had mysteriously disappeared from their e-book readers. These were books that they had bought and paid for—thought they owned.

Read more at Some E-Books Are More Equal Than Others. Apparently Amazon can delete books right off your Kindle without your permission. And yes, they did refund the purchase price of the book, but that is little recompense for their intrusion. The irony of it all is that the revoked book is George Orwell’s 1984.

I decided after using if for a while I liked it and could get used it using it. But I have also come to the realization that paper is a really ingenious technology. Suppose you want to spread out several reference works on your desk while you are studying? You can’t do that with a Kindle.

Paper is good.

Scripting Gnome

I wanted to automate some things on my Gnome desktop and being rather fond of Python, I thought that I would start there and see what I could do. My first task was to change the Gnome Panel from one screen to another automatically. I have a dual monitor setup and I want to automate the panel moving from one to the other.

I did a little research and came up with this little script that does the job. It simply toggles the panel between screen 0 and 1. I had no idea it would be this simple.

import gconf

path = ‘/apps/panel/toplevels/panel_0/monitor’

gc = gconf.client_get_default()
cs = gconf.ChangeSet()
cs.set_int( path, not gc.get_int(path) )
gc.commit_change_set(cs,True)

Sheepology Update

I finally started working on Sheepology again. I decided that the next order of business is a content management system (CMS). I have spent the last couple months working with Ellington and Jazbox here at the Naples Daily News and have developed strong feelings about these systems and consequently a vision for something better. Coupling this knowledge with my experience of Joomla and Bricolage, I think I have a pretty good idea of what a good CMS should be. (Bricolage is by far my favorite CMS, but not very friendly). And hey, Django was designed as a framework for building content management systems, so I’ve already got a good start.

In addition, the church I am attending has a strong need for a new website. I would like to use Sheepology to do that. Initially the integrated CMS will be simple so that I can get something out the door, but I do have a vision for something larger.

All that said, without thinking about it I packed away my desktop computer into my trunk to free up space on our camper’s kitchen table. Shari got a laptop and I decided that we no longer needed to have the whole desktop taking up all that space. So before I thought to commit my changes I tore it all down.

Thankfully we should close on “our” house by the end of July and all will be well.

Kindle 2

I have an opportunity to play with an Amazon Kindle 2 for a while. I’m not supposed to buy any books since it’s on the company card, but there are oodles of free books out there.

I’m a big fan of Christian Classics Ethereal Library CCEL, so I wanted to see if I could get any of their books onto this Kindle. It turns out that many if CCEL’s books are available in the online store for a nominal fee, but having been mandated not to buy anything I decided to see if I could go a different route.

I started out by downloading a PDF from CCEL and trying to convert that. That was not as easy as I thought it would be. You’re supposed to be able to send an email to a service on Amazon and have it converted, but apparently my address was not approved for that. So I installed Calibre. That actually worked pretty well. It converted the PDF to a .mobi file and then I was able to drag it onto the USB Kindle “drive.” Unfortunately, the formatting is really not that great.

It turns out that the easiest way to do this is to simply use a plain old .txt file. CCEL has .txt files for everything so I downloaded one (Eusebius’ Ecclesiastical History) and stuck it on the Kindle. Really bad formatting. No biggie because it’s just a .txt file. I can mangle it all I want with Python or Perl. So I put this little script together and it makes a very readable book on Kindle.

#!/usr/bin/env python
import re,sys

for line in file( sys.argv[1] ):
    if re.match('\s+$',line):
        print '\n'
    else:
        # shrink strings of spaces to one
        line = re.sub('[ ]+',' ',line).strip()
        # shrink strings of repeating characters
        line = re.sub(r'(.)\1{39,500}','\n'+r'\1'*40+'\n',line)
        print line,

My Mac

I’ve been using a Mac now for a week. This is the first time in my life that I’ve actually used a Mac for anything serious or for an extended period of time. I vowed to give it a chance for at least a couple weeks before I install Ubuntu.

The hardware is exceptional. I have a 17″ MacBook Pro with 4G ram and about 300G disk. The screen is beautiful, the touchpad is the only touchpad I’ve every gotten along with, and surprisingly the chicklet keyboard isn’t as bad as I expected. The aluminum case is a thing of beauty.

I am getting used to it. In fact, I could probably exist in the Mac world OK. That is to say that it is tolerable. Tolerable requires a good shell (that rules Windows out) and a good scripting environment including such things as Python and Perl. Those come bundled on a Mac. Tolerable also requires something like Spaces (something that Linux/Unix has had since the early 90s). While Windows doesn’t come bundled with something like Spaces, I know that there are add-ons that can do something similar. Tolerable requires easy access to tools like ssh, ftp, sshfs, etc. While that doesn’t quite rule Windows out, most of the Windows based tools that provide that functionality are sub-par and barely tolerable.

So yes, I can get along on a Mac. That does not mean it’s ideal.

I have 3 chief complaints with Mac OS. One, easy access to open-source software is poor. With Ubuntu almost any open-source package can be installed with a few clicks. Fink and it’s clones are sub-par. Two, it is relatively uncustomizable. That is to say I can’t set short-cut keys easily to do the things I want them to do; I can’t make the finder open files with a single click instead of a double; I can’t make the window focus follow the mouse without clicking (without buying a $15 add-on that still doesn’t work the way I want it to), etc.

And now for the biggest complaint, three, the basic premise of the UI is a throwback to the 80s. In the 80s desktop computers were a single-tasking deal. You could only use one application at a time. So at that point in time it made sense for Apple to put the application menu bar at the very top of the screen unattached to the application windows itself. That simply doesn’t make any sense anymore and it causes problems when you want to have things like focus follows mouse. Come on Apple, even Microsoft figured that one out.

So yes, I will give this Mac OS thing another week to see if I can somehow tolerate this broken UI, but I’ve already downloaded the latest Ubuntu in anticipation of my install next week. Supposedly this thing came with bootcamp…

First Day

Today is my first day at the Naples Daily News. It’s 88° here today.

Naples Daily News

I am currently in Naples Florida interviewing for a job at the Naples Daily News. Naples is what I would call a tropical paradise. It’s a bummer that there are no mountains for skiing or hiking, but it has its own beauty. Walking on the beach in shorts in March is pretty cool.

Sheepology Updates

I just finished updating the Sheepology demo with the latest code. New features include:

• fully function visual group tree editor
• email

The visual editor turned out to be a very good exercise in learning Flex. The editor is now fully functional. You can drag and drop groups onto each other to create parent/child relationships. Click on a connection to delete it. Click on the X in the upper right to delete a group. There is a button for creating new groups and a way to get back to the list view. Positioning is persistent. Changes made are immediately saved to the database. It is fully interactive.

After some thought, I concluded that this might be a useful tool for managing pages in a content management system as well and I may generalize it at some future point for that purpose.

The email system is rudimentary. You can send email to one or more individuals or to a group or a group and its subgroups. It uses TinyMCE for rich-text editing. It stores a history of any email sent. And automatically BCC’s the sender. I made a short detour from the visual group editor to make this happen because I thought it would be a necessity for any church wanting to use Sheepology in production.

The next thing I will be working on is some sort of reporting system as I think that will be critical. Other items on my list of future features are:

• inventory system
• a more sophisticated contact management system
• accounting
• integrated web content management
• on-line donations
• volunteer scheduling

I’m willing to takes suggestions at this point on what people would find most useful or necessary.

Sola Scriptura

I get in trouble in cycles. About 7 or 8 years ago I, through study, exited the Seventh-day Adventist church and entered the mainstream of protestant Christianity. This caused a major crisis in the Glass family. My spiritual steps go in cycles. I take a step and then settle into it for a while.

Over the last year or so I have been contemplating how to explain my thoughts on Sola Scriptura. It is not easy to explain. It is akin to trying to explain something in English to a person who only speaks Spanish. To a protestant it is as fundamental as gravity. It is so foundational to most protestant thought that protestant thought does not exist without it. It is the dividing line between protestantism and traditional Christianity.

The simple truth is that Sola Scriptura is a newfangled teaching. It was unknown to the Christians of the first century. It has only been around since Luther. It is not even supported by the scripture itself (a fatal flaw). So I was pleased to read a recent post by my favorite blogger Father Stephan. This post explains it better than I ever could.

What does this mean? It means that I can no longer in good conscience call myself protestant. What remains? The Roman Catholic West or the Orthodox East. That choice still lies ahead.

Upheaval

Having recently become unemployed, life is full of excitement. Thank God for the WARN act, which requires companies to pay employees for a full 2 months after announcing a company closure.

It looks very much like we will be relocating regardless of what job I take. For Sheepology that means development will be on hold for a couple months while we re-pack and relocate. Things are going to be very crazy around here for a while.

Final Edition

On Contemporary Christian Music

Last December I attended an evening service at a local church. During that service a young teen-aged girl sang a song I generally really enjoy – Breath of Heaven by Amy Grant. The young girl had a phenomenal voice and excellent control. She sang beautifully, but there was something wrong and I couldn’t put my finger on what it was.

In my head I could hear Amy singing it and I compared. There was something about the maturity in Amy’s voice that was missing in the present rendition and suddenly I knew what it was. There is something in the act of singing that you can’t really put your finger on but your soul feels.

I can’t tell you why I knew. I can’t describe musically what was different. But I could feel a difference in attitude and in the mental state of the singer. The young singer lacked humility. The most magical part about Amy Grant’s version of the song is that somehow she captures the humility present in the story of Saint Mary. The young girl in the present wasn’t singing for the glory of God with an air of humility. She was singing to impress the audience.

I have been bothered by most contemporary Christian music for some time now but I couldn’t put my finger on why. Electric guitars and drums are not what bothers me. Not all contemporary music lacks humility, but 90% of what I hear does. It is not about bringing glory to God, but glory to the singer.

Next time time you listen, don’t just listen to the words, the tune, or the rhythm. Listen to the soul of the singer. For whom does he sing? You’ll hear it in his voice and know.

Lord, teach me humility.

Sheepology Visual Group Editor

I have been madly working on a visual group hierarchy editing tool to incorporate into Sheepology. I wanted to learn Flex and this was a great opportunity to do so.

You can view my work in the demo on sheepology.org. Just go to the Groups page and click on the Visual Group Editor button. At this time it is read only. It does pull data from the database initially to draw the graph, but changes made are not saved. You can drag a group onto another group to create a subgroup connection. It is still very rudimentary.

Here are a couple interesting observations.

It is very easy to embed images into a SWF using Flex. I particularly like the fact that I can embed vector SVG images created in Inkscape.

I had a hard time getting Flash remoting via AMF working between Flex and Django because I didn’t want to use the add-on packages the documentation described how to use. Instead I slogged through making an alternative method work that requires no additional packages and actually ends up being just as simple. I believe it does rely on using Flex 3.0 or greater though. Note that the DjangoAMF AMF gateway is set to ‘/amf/’ and the remoting service name is ‘groups’.

// connect remoting gateway
import flash.net.NetConnection;
import flash.net.Responder;
private var gateway:NetConnection = new NetConnection();

private function initApp():void {
    if( Application.application.parameters.debug == 'True' )
        gateway.connect('http::8000/amf/');
    else
        gateway.connect('http:/amf/');

    initializeGroups();
}

private function initializeGroups():void {
    gateway.call( 'groups.allGroups',
        new Responder(
            function(result:Array):void {
                // success... do something about it
            },
            function(result:*):void {
                // failure... handle error
            }
        }
    );
}

Django Fun

I came across a cool Django tool called django-command-extensions that uses GraphViz to automatically generate a model graph of your application. Here is the graph I generated for Sheepology.

Check it out.

Sheepology Basics

There are a lot of details involved in the operation of a church. Those details will grow over time into Sheepology, but I wanted to identify the most basic fundamental structures and implement those first.

What I learned from experience volunteering in a church is that at the root of church structure are 3 things:

• People (individuals, families)
• Groups (ministries, serving teams, small groups)
• Events (Sunday services, small group get-togethers, team meatings, etc.)

This is how I organized Sheepology.

I’ve already talked about Groups, so let’s move on to People.

One of the key things I have done differently from most ChMS software I have seen relates to family membership. I had actually built this into my Perl-based check-in system several years ago and found that in today’s environment of divided families, it is very important.

A person can belong to more than one family.

This requires a slightly more complicated database structure (i.e. it’s a many-to-many relationship), but it is worth the extra complexity. There was one divorce situation I had to deal with repeatedly because the kids could not belong to both families in the ChMS. During check-in, the father would always attempt to use his phone number or name to check in the kids, but they would not come up. I had to explain that I had no control over the system and explain what information to use when checking in. I think he resented the fact that our database could not recognize his fatherhood of the children.

The other unique feature I wanted to add was an embedded webcam tool for taking photos of individuals. Try it out in the demo, it works. Additionally, if the individual does not have a photo, Sheepology will automatically attempt to use a Gravatar.

Eventually I would like to do something with Facebook integration.

Sheepology and Trees

First, I want to clarify what I said in a previous post. I said there are 2 ways to organize people, groups and tags. What I failed to mention is families. I originally considered making a family just a special case of a group, but I decided in the end that families are such a special and important group that it needed to be it’s own special construct. At one point I considered making Sheepology into a full-fledged genealogy system as well, providing ways to traverse trees of relatives, find cousins, etc. What I decided in the end is that it would be pointless to the operation of a church and just add unnecessary complexity.

On to trees…

The difference between a general graph and a tree is that a tree has no loops. The danger in allowing loops in your structure is that the methods I demonstrated in an earlier post could potentially run forever (though Python raises an error if you reach its maximum recursion depth, and of course you would eventually run out of memory). So in order to simplify things and actually have them make sense you need to prevent loops from occuring, ensuring that your structure always remains a tree.

So I extended my Group form to check for loops before validating:

  def clean(self):
    for g in self.cleaned_data['groups']:
      if self.instance == g or self.instance in g.getSubgroups():
        raise forms.ValidationError('Recursive Group structure detected.')

    return self.cleaned_data

Where getSubgroups() looks like:

  def getSubgroups(self):
    'Get all subgroups of this group.'
    for group in self.groups.filter( active=True ):
      yield group
      for subgroup in group.getSubgroups():
        yield subgroup

Sheepology and Church Structure

Many church management systems I have seen are what I would call unnecessarily complex. They start out with a special structure for ministries and then under that have special structures (and interfaces) for groups, sub-groups, staff groups, etc. The interface becomes unwieldy, unintuitive, and basically requires a manual sitting next to you to use correctly.

To top it off, if you want to make a report of who showed up to what event and belonged to such-and-such a sub-group of X ministry and Y group, you almost have to write a custom report. This tends to produce a system in which you basically need a Google-level search engine just to find a report that will work in your situation.

Sheepology has 2 ways of organizing people: Groups and Tags. Tags you all know and love. They are a quick and dirty way to organize things. The danger in tags is that you can get too many words that mean the same thing and your staff have to voluntarily comply with a standard tagging scheme if you want to use them to organize your structure. Tags are best as an informal quick-and-easy way to visualize your body and keep track of nice-to-know aspects of an individual (or group), i.e. skills like cooking, skiing or running.

What all the different layers of structure in an organization boil down to is a classic tree structure. You have nodes and connections between nodes. Why make it more complex than that?

Basically a Sheepology Group contains people and other Groups (and of course meta-data like name and description). You can have a tree that goes down 42 levels deep if you want. You can have as many top-level nodes as you want. You can basically have multiple inheritance if you want. One Group can be contained by as many parent Groups as you like.

So suppose I want to find all the subgroups of a particular group? The Group class has a simple recursive method called getSubgroups. This method returns a python generator  (i.e. it acts like an iterator):

    def getSubgroups(self):
        'Get all subgroups of this group.'
        for group in self.groups.filter( active=True ):
            yield group
            for subgroup in group.getSubgroups():
                yield subgroup

Or suppose I want to find all events that members of group X can check into.

    def getCheckableEvents(self, **kwargs):
        'Get all events for which people in this group may be checked in.'
        for event in self.events.filter( checkin=True, active=True, **kwargs ):
            yield event

        for group in self.parent_groups.filter( active=True ):
            for event in group.getCheckableEvents( subgroup_checkin=True ):
                yield event

Sheepology and IE

BTW, I don’t even know if Sheepology works in IE. Haven’t bothered to check. For now use Firefox.

More on Sheepology

At the Rocky, we were beginning to create a number of geographically oriented applications (see our Holiday Lights). Lucky for us, GeoDjango had been recently rolled into the official release of Django. By simply installing the PostGIS extension to PostgreSQL, we had built-in geographic proximity search and much more. Here’s a sample that finds all addresses within 5 miles of the current address:

nearby = Address.objects.filter( point__dwithin=(self.point, D(mi=5)) ).exclude( id=self.id )
if len(nearby) > 0:
raise SomeException('Uh Oh...')


This type of feature not only makes it easy to find people who live near each other (presumably by small-group staff), it allows for improving data quality. When storing an address in the database, you can check to see if there is a likely duplicate (i.e. an address that is within a meter or so of another address) and prompt the user for an override if necessary. I have built this into Sheepology.

Something I have always been interested in is AI and natural language processing. So I was somewhat interested in making searches more fault tolerant. One problem we had at Crossroads was that we ended up with a lot of duplicate people in our database because searches failed. I wanted to improve the situation. I did a little reasearch and discovered an algorithm called soundex. While many databases have this built in, they do it on the fly, so doing a search becomes very slow. I decided to simply make an extra field that contained the soundex codes for people’s names. This way I am able to index the field and make the search fast. Python has a module called AdvaS that has the soundex algorithm and several other similar algorithms. So in just a couple lines of code I was able to have phonetic search. Try it in the demo if you want. Search for bryan and you’ll find brian as well. Or search for sherri and you’ll still find shari. In the main settings file you can choose your phonetic algorithm, but it defaults to soundex.

There is also a nice tagging module available for Django. It was remarkably easy for me to plunk this in and magically have tagging enabled on models I chose. In about 5 lines in my template I was able to create a tag cloud.

I had originally decided to develop my own UI for Sheepology, but decided to work on building the basic models first. What I found is that the built-in Django admin facilities are so extensible, that I may stick with the built-in admin rather than building a UI from scratch. I also added the Grappelli theme to jazz it up a bit. While I’d like to take credit for the look of the admin, for the most part it is a packaged theme designed by someone else.

What I have found with Django and Sheepology is that I am most effective as a systems integrator. While I have written a fair amount of custom code, what I am really doing is tying together a number of different open-source pieces to make a whole package. I can focus on my area of expertise (i.e. churchy stuff) and worry less about details.

I was able to build what you see as a one-man team in about 2 months. What I see down the road is nearly limitless. There are a number of Django open-source packages for doing such things as content management or accounting that could potentially be integrated into Sheepology for a comprehensive ChMS package.

On the immediate horizon I will be working on a graphical group hierarchy tool. I also see a need for an inventory tracker (portable churches desparately need this) and contact management.