Shift8 Creative Graphic Design and Website Development

javascript

MongoDB: Dealing With Data That Gets Confused as Sub-Objects

Posted by Tom on Wed, Apr 04 2012 09:00:00

So I came across an interesting challenge the other night. I want to store in MongoDB a bunch of URLs and how often they are accessed. Call it simple metrics for a web site. The structure for my JSON response that I want is like so:

{
'http://www.site.com/whatever/page.html' : 23,
'http://www.site.com/another.html' : 4
}

The problem is saving this into MongoDB field isn't possible in this structure. If you don't know why, it's because MongoDB will see periods as an indicator for sub-objects. So if I go to save that under a "urls" field in my collection I'll end up with { http://www { site { com/whatever and so on.

encodeURIComponent? Sure, but periods don't get encoded to ASCII equivelant of %2E and don't need to according to RFC. Even though we could probably replace them, what if we don't want to end up with % symbols everywhere? What if we have, instead of URLs, geo coordinates? Lat/lon pairs that are not strings would not pass through encodeURIComponent. We would need to cast as string and send through, etc.

Why not add slashes or other characters? Because, if you are dealing with URLs, those characters could be mistaken for valid. There's going to be limited options for you to replace a period.

So encodeURIComponent plus some replace magic is one possible solution (didn't really do it for me though). I prefer to base64 encode the values. Of course base64 isn't native to JavaScript, but thanks for PHP.js we have some pretty sweet functions ready to use. See here for a base64_decode() equivelant to PHP's. Of course, both the encode() and decode() functions for base64 also require a UTF8 encode/decode function. So, there's 4 functions in all that you'll need.

Here's what the values look like stored in MongoDB now (for a hypothetical "urls" field):

'urls' : {
'aHR0cDovL3d3dy5zaXRlLmNvbS93aGF0ZXZlci9wYWdlLmh0bWw=' : 23,
'aHR0cDovL3d3dy5zaXRlLmNvbS9hbm90aGVyLmh0bWw=' : 4
}

So how do you use this in MongoDB? Simple, you can place the code in any map/reduce or finalize MongoCode. You probably don't want to keep doing that over and over though in each of your files and if you work from the command line, it'll be a nightmare. So you can also save stored JavaScript in MongoDB! Then you can simply call the function as if it was native.

Here's a few sites with further reading on stored JavaScript:

Now when you run aggregation, say a group() query, you can decode the values back. You could also decode the values in any other language that has base64 decode capability like PHP. You could keep the PHP.js functions on the front-end and let someone's browser do the work as well.

How much time does it take for all this? What's the overhead? Well, I haven't benchmarked it extensively. I only benchmarked my aggregation process, but I can say that it didn't take anymore time really. We're talking fractions of a second. Admittedly the job only took 2 or 3 seconds anyway, but regardless if I was running the encode function or not it was the same time. I imagine a much larger job would see a noticeable difference, but also keep in mind that a larger job is taking time anyway. So if you're concerned about using this function for a query that you want to happen without a page load timing out...Don't.

I'm also interested in seeing if there's also compression functions that can be used to save data. LZW compression has been implemented in JavaScript and the patent has apparently run out on that so it's kosher to use. Keep in mind that base64 requires about 33% more space for the data. If you're trying to keep an effecient document size, it may not always be the answer. However, the values are definitely key safe and I imagine any other kind of encoding to avoid periods also adds size.

Typographic Lockups

Posted by Tom on Fri, Sep 02 2011 13:38:00

type lockupI haven't designed in a while. It's just been too crazy busy with programming web sites. I'm working on a project now where I get to design again! It's very exciting. I realize now I have a real good ability to see both sides of the fence here. I'm an educated graphic designer and took many years of typography classes (more than the required amount). So I know exactly what I'm doing with regard to setting type and I also happen to love it. Sadly, I settle for what the web has and it's limitations. I also know that if I go and create images for every single bit of text...I miss out on SEO and it's a real pain to update. 

So we suffer when it comes to the web. We all do. Of course there's various forms of dynamic text replacement, some using Flash, others PHP (with GD library), and then of course don't forget the new possibilities with CSS @font-face. FontSquirrel and Google are two really good places to get some free fonts that are more than your basic web fonts.

That's great and all, but... We're still limited. Being creative takes a lot of work manipulating CSS and HTML. However, it is possible. The problem is that it's pretty much single serving. You don't always know how many words are in a piece of copy so it's hard to create re-usable styles for things like titles, subtitles, and pullquotes. For example, look at the titles for my blog posts. Sometimes there's enough words in the title that it spans several lines. When you compare that against setting type for print... You realize that you can adjust things to make it fit on one line if so desired. On the web, we have to live with what we get for the most part.

I'm trying to fix that. I'm working on a jQuery plugin that allows you to create beautiful typographic lockups and have those be a re-usable set of rules. So you can apply the same style to, say, all your blog post titles, but with a bit of intelligence. Things like if there are this many words do this, or break the line at the 5th word, etc. Think of it like programming working with design in unison. It's beautiful. See (and see above)?

typographic lockup

Now, the best part is that this lockup works with 4 or more words. It'll just keept wrapping if more lines are needed. Every word including and after "post" in this case will be the same size and everything. The only thing required to output the above is a span element wrapped around the title. So when it comes to something like creating a template for a CMS or blog, there's no extra work beyond applying a class.

How does it work? Well, in the above screenshot, both the date and the title have a span element wrapped around them with a specific class. The jQuery plugin then basically applies a set of "typographic rules" to the copy. Things like how many words per line, etc. The date is slightly different, that just gets broken out so that the day, month, year, all live in their own span element. It's much more straight-forward than the type lockups.

The JavaScript then breaks up the copy and replaces it with new HTML that has each line wrapped with a div and each word wrapped with a span element. Classes are also present with all this marking each individual word and line so they can be targeted exclusively with CSS. For example: l_1, l_2, etc. for lines and w_1, w_2, etc. for words. Each line and word also have an "l" and "w" class on them allowing for a broader set of CSS rules. Of course, each lockup can have its own class and id values applied to it so you can further target specific lockups to apply different styles.

Once you get your CSS defined and what I'll call the "sequencing" of words per line, you're set! You can then copy and paste the styles and JSON settings for the jQuery method call and share them across sites and such if you want. It's just a tiny bit of configuration there to create beautiful and re-usable typographic lockups.

I'll be working on this a bit more to polish it and add more settings. I also have an option to wrap everything in quotes if you want as well for things like pullquotes. You'd simply add a "q" class to the element you applied the method to or you'd pass to the method an option for quotes set as true. 

More to come soon!

Agile Uploader Now on Github

Posted by Tom on Mon, Aug 01 2011 21:10:00

Agile Uploader is now available on Github! I've decided to release the source code for the project after a long while and careful thought. The reason? I simply can't continue to maintain it. The tool does exactly what I was after and quite well for my needs. While I still will use other upload tools, when it comes to needing to resize images that users upload (namely photographs, typically off SD cards), it does the job perfectly.

However, the tool was falling short of some (maybe 10%, or less, of the comments I received) user's needs. I briefly went over some of my research involved with building the tool before, but I don't really want to make anyone work hard to get from A to B. So I'm going to give everyone B and if you want to get to C...Then, by all means, go for it. Smile

You can find the project here on Github. Please fork it and feel free to leave feedback. I can't promise that I'll be working on it much more, but I will fix any major bugs. I hope that people can add features that will help others out there.

A note for what you're looking at...You are looking at a FlashDevelop project. In the normal "bin" directory, where the SWF file is output to by default, I also included all the demos for the tool. Most importantly is the agile-uploader-3.0.js file. This is the jQuery plugin that I built and bundled with Agile Uploader. This directory structure leaves a bit to be desired I know, but it should give you everything you need. Please reference the jQuery plugin along with comments in the source code of the Main.as file. The tool really works by Flash's ExternalInteface communication of Flash <-> JavaScript. You don't need to use jQuery, but you'd need to write your own JavaScript to use Agile Uploader if you didn't...Or if you wanted to write your own custom jQuery too of course.

You can work on Agile Uploader with the Flex SDK and FlashDevelop. The tool is built completely upon open source code.

Enjoy!

Map/Reduce in Lithium for Visitor Metrics

Posted by Tom on Tue, May 24 2011 10:01:00

Update: So after I got a little further along with this very example in a real life project I realized that while it makes for a very simplistic illustration of map/reduce (that I personally found helpful when learning how to perform map/reduce), it is not a very good real life example. The reason being... Smile ...The document size limit in MongoDB. Doh! You couldn't store metrics like this. However, ignoring the purpose of this, you can still continue reading about how to perform a map/reduce within Lithium.

Original Blog Entry

I'll start off by saying I love MongoDB and map/reduce after putting it off for some time. I dreaded learning the map reduce functions big time. It turns out, it's not that bad. A friend asked me to explain it in 10 words or less. So I did. It's not really all encompasing of the features, but it's a real good example for what map/reduce can do for you.

Use JavaScript to identify/"map" data to loop it to aggregate/"reduce."

Ok, so that's 12 words technically, I cheated by adding slashes and combinging two words. It's also really poor grammar. Anyway, that's the idea. In this example, I wanted to collect information about visitors on a web app. Obviously I'm not a masochist, I'd use Google Analytics if I could...Sadly, I could not. So what to do? Well, we can use MongoDB to record all this data and then use map/reduce to get some totals.

I may eventually turn this into a Lithium library (especially because there's a good browscap and language detection class that I'm not illustrating here), but for now I'm going over things at a high level and focusing on the actual map/reduce process.

That said, imagine a data set like this:

"metrics": {
    "pageviews": 63,
    "visitors": {
      "192-168-126-1": {
        "ip_address": "192.168.126.1",
        "browser": "Chrome",
        "browser_major_version": 11,
        "operating_system": "Win7",
        "mobile_device": false,
        "primary_language": "en-us"
      },
      "192-168-126-2": {
        "ip_address": "192.168.126.2",
        "browser": "Firefox",
        "browser_major_version": 4,
        "operating_system": "Win7",
        "mobile_device": false,
        "primary_language": "en-us"
      },
      "192-168-126-3": {
        "ip_address": "192.168.126.3",
        "browser": "Chrome",
        "browser_major_version": 11,
        "operating_system": "Win7",
        "mobile_device": false,
        "primary_language": "en-us"
      }
    }

Now, we have this "metrics" field where ever you like, but in my case on a document that contains some other information. Why not a separate "metrics" collection? We could and then we could also put in things like page URLs that were hit on the site to start getting analytic information about the pages on our site. In my case, I just wanted to get a sense for some high level information about my visitors. For now.

So the first thing here that you'll notice (and I've written about the $set operator before) is that each IP address is the key for each entry. The dots have been replaced with dashes so that it works as a key. Otherwise, I'd have a pretty deep object on my hands. Surprised

So each time a page is loaded the pageviews count goes up and the visitor's browser information is captured using $set so that if the user from the same IP address came back again with a different browser, it would update. My metrics would not be skewed. Yes, it's sad that we don't realize when/if the user actually uses two different browsers...More sad that we're likely counting entire office buildings as one user, but that's just how the cookie crumbles in this case.

Ok, so we have that data and we have some controller action in our Lithium project that's going to return to us an array that we'll pass to the view template to make some pretty pie charts. Why not pie charts? I love pie charts, they give everyone a sense of satisfaction that looking at numbers is really fun! ...Or something like that.

We'll dive right in. Here's the entire action I'm using with the map/reduce code. Note that Lithium's MongoDb adapter does not have any options for map/reduce in the find() or any other method. I may write something in the future for that myself if I end up doing enough of these (and I likely will). However, we can make straight up command() calls from it.

 public function metrics($url=null) {
        if(empty($url)) {
            return false;
        }
        
        $db = Project::connection();
        
        // construct map and reduce functions
        $map = new \MongoCode("function() { ".
            "emit(this.metrics.visitors, this.metrics.visitors);".
        "}");
        
        $reduce = new \MongoCode("function(k, vals) { ".
            "var visitors = vals[0];".
            "var unique_visitors = 0;".
            "var b_counts = new Array();".
            "var browsers = new Array();".
            "var os_counts = new Array();".
            "var operating_systems = new Array();".
            "var mobile_devices = 0;".
            "var ln_counts = new Array();".
            "var languages = new Array();".
            
            // loop all the emitted visitor metrics to aggregate some data
            "for (var i in visitors) {".
                // count browsers
                "if(typeof(b_counts[visitors[i].browser]) == 'undefined') {".
                    "b_counts[visitors[i].browser] = 0;".
                "}".
                "b_counts[visitors[i].browser] += 1;".
                
                // count operating systems
                "if(typeof(os_counts[visitors[i].operating_system]) == 'undefined') {".
                    "os_counts[visitors[i].operating_system] = 0;".
                "}".
                "os_counts[visitors[i].operating_system] += 1;".
                
                // count the primary languages
                "if(typeof(ln_counts[visitors[i].primary_language]) == 'undefined') {".
                    "ln_counts[visitors[i].primary_language] = 0;".
                "}".
                "ln_counts[visitors[i].primary_language] += 1;".
                
                // count the number of mobile devices
                "if(visitors[i].mobile_device == true) {".
                    "mobile_devices += 1;".
                "}".
                
                // count the number of unique visitors
                "unique_visitors += 1;".
            "}".
            
            // loop browsers counted and set for output
            "for (var x in b_counts) {".
                "browsers.push({ name: x, count: b_counts[x] });".
            "}".
            
            // loop operating systems counted and set for output
            "for (var x in os_counts) {".
                "operating_systems.push({ name: x, count: os_counts[x] });".
            "}".
            
            // loop languages counted and set for output
            "for (var x in ln_counts) {".
                "languages.push({ name: x, count: ln_counts[x] });".
            "}".
            
            // return the output
            "return { 'browsers': browsers, 'operating_systems': operating_systems, 'languages': languages, 'mobile_devices' : mobile_devices, 'unique_visitors': unique_visitors }; }");
        
        $metrics = $db->connection->command(array(
            'mapreduce' => 'projects', 
            'map' => $map,
            'reduce' => $reduce,
            'out' => array('merge' => 'mapReduceMetrics')
        ));
        
        $cursor = $db->connection->selectCollection($metrics['result'])->find()->limit(1);
        foreach ($cursor as $doc) {
            $results = $doc['value'];
        }
        
        // Get the total page views for this project
        $pageviews = Project::find('first', array('fields' => array('metrics.pageviews'), 'conditions' => array('url' => $url)));
        $results['pageviews'] = $pageviews->data('metrics.pageviews');
        
        return $results;
    }

Yea, it's not the prettiest to look at. It's my first run through and it's literally based off an example from php.net so that's why there's all those lines concatenated together like that. I wouldn't normally do that. Nor would I use heredoc...But something a little nicer, at least single quotes instead of double. Anywyay, with that you will be returned a nice array (in $results) that will show all the counts for browsers and such. Note, I did not take into account the browser major versions here in this example. Also note that I separately stored a pageview count on the document which does not require a map/reduce to retrieve. 

Now let's look at it deeper. There's a lot of good articles on map/reduce if you spend time with them, they should be pretty clear. Here is a good one. Then you can also look at the MongoDB Cookbook site's example. Also php.net's example. You'll see that you can use map/reduce for many things. Let's go over how I'm using it.

First, the map function. Pretty simple. In fact, you likely wouldn't do what I'm doing here. The idea of it is to basically grab keys and values for a given collection. Those keys should be unique. So in my case metrics.visitors are unique keys. They are also the values that I need. What this does is returns the values to a reduce function.

The reduce function. More complex, but it's all nice friendly JavaScript. Here you're just looping the values that are passed and simply counting some of them. As a disclaimer, my example could have probably been written a lot better and cleaner. I only loop once which is what I was concerned about mainly. The rest can be refactored later.

At the end of whatever you decide to do with all that data, you'll return your values. I'm returning an object here with all the counts. Here's what PHP gets back in $results:

array
  'browsers' => 
    array
      0 => 
        array
          'name' => string 'Chrome' (length=6)
          'count' => float 2
      1 => 
        array
          'name' => string 'Firefox' (length=7)
          'count' => float 1
  'operating_systems' => 
    array
      0 => 
        array
          'name' => string 'Win7' (length=4)
          'count' => float 3
  'languages' => 
    array
      0 => 
        array
          'name' => string 'en-us' (length=5)
          'count' => float 3
  'mobile_devices' => float 0
  'unique_visitors' => float 3
  'pageviews' => int 256

...And there ya have it. What I would do next is actually cache this data so each time I called the action, it didn't have to run the map reduce which could be quite expensive over time with a lot of data.

Cool note: In this example you see the $metrics = $db->connection->command(...) part? Run a var_dump() on $metrics. It will have some handy information for you. It could tell you about an error when it comes to parsing your functions (though I'm not sure how to actually debug things, sorry). It also will tell you if everything was ok and ran successfully. You may wish to check this before returning data. It's on my to do list myself. Also, it will show you how long the operation took which is very handy. You might need/want to index some fields and cache results based on how long things are taking.

Another note: With map/reduce you're actually outputting to a collection. So you're going to pick up your results with another query to that (temporary or not so temporary) collection. This changed in MongoDB version 1.8.0. You now have to specify that 'out' key in the command() call. Here's more information on that

Hopefully these snippets will be of some help to people. I didn't want to go too far in depth with explaining everything, I think there's other really good articles on that out there. My hope is that seeing an example, as it works within the Lithium framework, will be helpful.

Agile Uploader 3

Posted by Tom on Sat, Jan 08 2011 12:50:00

I'm pleased announce, after a few days of really focused work, I've released a new version of Agile Uploader. For those who haven't seen it before, it's a Flash client side resize before upload, multiple file upload tool. Why another multiple file upload tool? Well, swfupload and flash file uploader and uploadify and plupload and all the others (too many to mention) work great and all (and believe me, their authors do a great job and it's not easy and the tools do work well), but they didn't quite do what I needed. First off, several of those I just mentioned don't have client side resizing for image file types. What I mean by that is Flash can scale and re-encode new jpeg files that can then be uploaded to your server rather than having your server do the resizing. Obviously this helps with hosting costs, site scalability, and for those using shared hosting. Second, Agile Uploader is fairly lean and extremely customizable. Agile Uploader allows you to customize how things look and function with regard to what type of files you allow and how you want it to resize images (if at all).

Agile Uploader ScreenshotAgile Uploader asynchronously resizes and encodes images so you can keep attaching more files while it's working. The tool has a bunch of other options as well as a tight communication with JavaScript. I wrote a jQuery plugin to accompany the tool, but you could write your own JavaScript should you choose (or alter the jQuery plugin). Flash will pass to a defined JavaScript function various event data as it does its job. In fact, you can control just about every facet of the upload tool right from JavaScript. The only thing you can't use JavaScript for is the "browse" or "attach" button, it's against ActionScript's security rules sadly. However, you can customize the button that appears in the swf by specifying the path to your own image(s). You can also send the form via JavaScript if you wanted to automatically or on some event (like a button or link click). 

Documentation for this new version is on the way, there's been a lot of changes since version 2.x. It's become a lot more organized and simplified now. One of the long awaited features includes the ability to select multiple files at the same time. While the tool always uploaded multiple files at the same time, the user had to click the button to browse for each file one at a time. So the user experience has improved quite a bit along with a few minor bug fixes. 

You can see more about this tool, and a working demo, on the Agile Uploader project page.

Decompression: Discoveries and Current Projects

Posted by Tom on Tue, Oct 26 2010 19:35:00

So I want to make sure that I keep posting content on my blog. I not only want people to come back to my site, but I want to get in a good habit of writing as well as make sure that I'm jotting down some of my thoughts. The amount of crap that runs through my head is probably more than the average person's. That's not a pat on my own back, that's actually quite sad because I just want things to turn "off" sometimes. Sleep deprivation, lack of focus sometimes, and overall insanity is really what it leads to. Sticky notes and endless reams of printer paper, and sketchbooks (not that I sketch anymore like I should) really aren't cutting it. I need to decompress on my blog as well. So I'm adding some new categories to help stay organized.

Believe it or not, I use my own blog for reference. I do come back to what I wrote down and use it to copy and paste code snippets and keep tabs on where I was and where I am now. Sometimes I rant sure and those posts may not do any good for anyone...But I'm bored, and probably angry at the moment. Or, maybe I'm procrastinating. Like I am now...It's about the middle of the evening and I should be working on my little lightweight CMS but instead I'm writing...Hmm...Oh well.

So in the spirit of keeping it interesting for you all, check this out! Have an Android phone? Remotedroid can be found in the app market, but the server can be found on its Google code site. It's cross-platform, just run the jar file. Make sure that you have port 57110 open for UDP traffic (check computer firewall and router). 

You can hold down on the track pad area with one finger and swipe the other up and down and it should scroll. It may support some other gestures too, I haven't tried. It works really well as a mouse. The keyboard I found to be a little slow, but bearable for simple things like surfing the internet or perhaps your media center computer (ie. Boxee, etc.). That's really what I intend to use it for. Sure, there's a Boxee remote app for nearly all phones, but this works much better in my opinion...Especially if you want to do more or Boxee crashes or you don't run it all the time. Boxee seems to get a weird resolution change issue after my computer wakes up...So I have to exit it and restart it. Can't do that with the remote app. However, the remote app does have a novel picture of whatever is playing.

Anyway, a nice little discovery. What else? Well, just boring stuff. Things that I'm working on as I mentioned...A lightweight CMS. It's coming along well. It was the basis of the previous post about including external JavaScript files from another JavaScript file. This lightweight CMS doesn't use any framework...Or database even. It's intended for very basic, static, sites. Old sites. Imagine those sites for small businesses that were designed a while ago, or not necessarily a while ago, but perhaps designed very statically...This is quite common actually even in a world where we have Wordpress and Joomla and Drupal and Croogo! Don't forget that nice CMS. One day add Minerva to that too. Another project of mine for those of you who want to laugh, go ahead...But I promise that one will get finished as well.

I'm calling this lightweight CMS "Argos" with the metaphor/slogan of "Who says you can't teach an old dog new tricks?" So these very simple (1-20 page) sites are really the target. You can't use Argos on a dynamic site. Argos actually writes directly to the HTML or PHP page and alters the HTML contents of it. It of course stores data in JSON files to keep backups (also for historic rollbacks) and also backs up the entire site on installation (well, it will when I'm done). This ensures that the site using Argos doesn't get messed up and also helps to prevent user mistakes...Or rather, allows a user to "undo" things...Something I think that's missing from many CMS' out there. It's also designed to be quite compatible. Basically PHP 5 is the only real server requirement. It has to have the JSON PHP extension. That's the extent really of the requirements. Of course many shared hosts do offer MySQL but I figured I'd keep it as compatible as possible. Plus, do I really want to setup a database? What if I'm not installing the CMS? I want to deliver it with one PHP script. Let it download the files it needs and walk the user through installation. I'm assuming that they don't know what MySQL is and they certainly don't know how to setup a database and then a user to access said database.

The CMS does have a backend, but it's very simple for now. It does include a nifty file manager script that I found. So, there's another great find for you all...phpFileManager. It's just one PHP script actually, it works really well! I was thinking about expanding upon it and adding a few features and then just using it as a "swiss army" knife for web development. Adding things like markItUp! to it and so on...But keeping it all one file. It doesn't matter if it ends up being a few megabytes even...Just being able to get onto a server, wget the file from my server or FTP it somewhere, then load it up to go work on something in a pinch would be great.

Anyway, enjoy the discoveries and updates. I do post minor things like this on my Twitter feed by the way, so follow me!

Including External JavaScript From Another JavaScript File

Posted by Tom on Sun, Oct 24 2010 10:53:00

So I was looking around the internet for a way to include external JavaScript files in another JavaScript file. After a little bit of searching, the goal was clear...Use AJAX (XMLHttpRequest) to pull in the contents and then run evil on it. Err, I mean eval()... I definitely feel like it's wrong and against all the good rules we know, but it works and I think it's the only logical way to do it. Why on earth would one ever do it? Well, typically you wouldn't. Typically you'd just include your scripts on the HTML page...However, in a current project for a client I wanted to include one JavaScript file...Yet include jQuery and other scripts in that file. The reason being -- simplicity. It won't be me that will be including the JavaScript on the page(s) of the final site. So I don't want the (perhaps non-savvy) person to have to include all these scripts.

Reason number two. I don't want all this JavaScript loading for every person that hits these pages. It's actually a tool to maintain content on the site (let's just say a lightweight CMS). These tools are for editors only, despite that it won't do anything harmful if anonymous visitors hit the pages, it's still something that normal people don't need. It would technically slow down the page load time. However...Not if the embedded script is small and it selectively (based on "who" is loading the page) loads these external scripts. What if the site doesn't use jQuery? Why include jQuery and all this other stuff just so the client can edit the page? That sounds silly. Why is the JavaScript needed in the first place? Well, the CMS uses it for "edit in place" abilities and I have to assume the page is an HTML file so JavaScript would be the way to go. 

So eval() isn't quite evil in this case...It's actually our saving grace. Of course I could have made a "bookmarklet" for the editor to drag onto their page which would load up the editing tools. This would eliminate the need for any script to be included on the page and the normal visitor wouldn't be able to tell that there was any CMS powering the site (except for perhaps some subtle clues in the div tags)...But, I don't think bookmarklets are that great of a work flow. I don't think they are obvious enough for the non-savvy user. They are a neat trick sure enough, but it's an awkward process. They are a work around themselves...So what's the difference?

So then, how does one include external JavaScript files from another one? Like so:

    
function IncludeJavaScript(sURL) {
    var oRequest;
    
    // if using a normal browser
    if (window.XMLHttpRequest) {
        oRequest = new XMLHttpRequest();
    } else {
        if (window.ActiveXObject) {
            // if using IE 6 for some terrible reason
            oRequest  = new ActiveXObject('MSXML2.XMLHTTP.3.0');
        }
    }
    
    oRequest.open("GET",sURL,false);
    oRequest.setRequestHeader("User-Agent",navigator.userAgent);
    oRequest.send(null)
    
    if (oRequest.status==200) {
        eval(oRequest.responseText);
    } else {
        // alert("Error executing XMLHttpRequest.");
    }
}

// Now call the function, passing the script, here I'm including jQuery
IncludeJavaScript('/js/jquery-1.3.2.min.js');

Note that you will want the 3rd parameter of open() to be false so that the data loads synchronously. If it was asynchronously loaded, the rest of the code would execute and you'd probably have some undefined function errors and things not working. Another thing to note is that the external JavaScript MUST be on the same domain. JavaScript has a cross-domain security restriction. This is a good thing. You may be able to get around this though by having say a PHP "proxy" script on your server that grabs the contents of an external URL (the target JavaScript) and simply prints the results on the page.

Believe it or not, I found many incorrect answers to this problem out there so I hope reposting some of this research is helpful to someone else. I've tested this in Chrome and Firefox. I'm assuming it works in others too...Well, at least a modern version of IE.