Tomer Doron's Technology Blog

google style gauges using d3.js

2011-12-19T15:37:00.000-08:00

I’ve been playing lately with SVG visualization using the excellent d3.js library. I've originally chosen d3.js because I needed to create highly customized visualizations and d3.js is a great tool for dealing with lower level visualization.

some of the visualizations included gauges. In many of my products I use google's gauges, but in this case I needed to support offline mode, and google's charts require an internet connection. so I decided to go ahead and rebuild it using d3.js.

below are links to the code and example

http://bl.ocks.org/1499279
https://gist.github.com/1499279

note: the transition effect of the "pointer" still needs love, feel free to fork or otherwise contribute.

a less simple safe-html sanitizer

2011-03-03T21:35:00.000-08:00

Safe-html has been promoted by google and others as a solution for xss, specifically when dealing with user generated content. Unfortunately GWT provides a rather naive implementation of an html sanitizer named SimpleHtmlSanitizer which I found too simple for even simple use cases. Relying on the GWT framework and modeled after the SimpleHtmlSanitizer, here is what I came up with https://gist.github.com/1499453

fun with PPC

2011-02-05T20:28:00.000-08:00

I recently received an old Apple G5. Even after a decade its dual 64bit PPC CPUs and 8GB of RAM make it quite a capable machine. So when I sent my primary box to the lab, I decided to spend a day on setting it up as a development workstation.

With Apple not supporting the PPC processors line since Snow Leopard, I decided to go with a Linux setup. I tested a few alternatives: Fedora 12, Yellow Dog, Ubuntu and Debian Lenny and Squeeze. Eventually choosing Squeeze as it offered good hardware support and a more minimalist UI. Too bad CrunchBang does not support PPC, it is by far my favorite distribution for a desktop. Overall, installation of Squeeze was a breeze, more info can be found here.

My primary programming languages these days are Scala for the back end and GWT for the front end, so the next step was getting them to work.

Scala
The latest Scala package on debian repositories is 2.7.7, so you will need to download the latest from Scala's site and configure appropriately. So far so good, but a simple test yields bad news, Scala was super slow. After further digging I learnt that the root cause was the JVM: Squeeze comes pre-packaged with OpenJDK which is extremely slow to the point of unusable on PPCs as it is running in interpreted mode. Luckily, there is a simple solution, installing IBM JDK found here. Make sure to download the 32-bit if you are using a G5 and that libstdc++5 and libgtk1.2 packages are installed

GWT
To develop for GWT you need to have Eclipse with the GWT plugin installed. This part is easy, just make sure you download the 32bit version of Eclipse if you are using a G5. The troubles begin when you are finally ready to debug: GWT debugging is dependent on a browser plugin that unfortunately is not supported on PPC Linux. Luckily, the GWT code is open, so with jumping a few hoops, you can make it work using the following steps:

1. make sure xulrunner and xulrunner-devel packages are installed

2. download GWT source:
$ svn checkout http://google-web-toolkit.googlecode.com/svn/trunk/ trunk
$ svn checkout http://google-web-toolkit.googlecode.com/svn/plugin-sdks/ plugin-sdks

3. copy the plugin SDKs from the x86 version as a PPC version:
$ cp -R plugin-sdks/gecko-skds/gecko-1.9.1/Linux_x86-gcc3 plugin-sdks/gecko-skds/gecko-1.9.1/Linux_ppc-gcc3
$ cp -R plugin-sdks/gecko-skds/gecko-1.9.1/Linux_x86_64-gcc3 plugin-sdks/gecko-skds/gecko-1.9.1/Linux_ppc64-gcc3

Note that gecko-1.9.1 maps to firefox 3.5, so if you are attempting to compile for a different version you will need to change the path accordingly. Read the make file for the complete version numbers mapping.

4. Replace all the files in Linux_ppc-gcc3/lib, Linux_ppc-gcc3/bin, Linux_ppc64-gcc3/lib, Linux_ppc64-gcc3/bin with the "real" ones from your system, they are all found either in /usr/lib/xurlrunner-devel-1.9.1/sdk/lib or in /usr/lib.

This assumes you running xulrunner version 1.9.1, otherwise, your path may differ.

5. prepare to compile:
$ cd trunk/plugins/xpcom
$ export BROWSER=ff35
$ export DEFAULT_FIREFOX_LIBS=/usr/lib/xulrunner-devel-1.9.1/sdk/lib/

Again, this assumes you are running xulrunner version 1.9.1 and trying to compile for Firefox 3.5, you will need to changes these if you system is running different versions.

6. edit the install-template.rdf file, adding an entry for Linux PPC after the other platform entries:
<em:targetPlatform>Linux_ppc-gcc3</em:targetPlatform>

7. finally, compile GWT from source
$ make clean
$ make

This will create a Firefox plugin named "gwt-dev-plugin.xpi" in the prebuild directory, install it using Firefox.

integrating elasticsearch and lift

2010-11-06T03:02:00.000-07:00

I recently came across elasticsearch and was waiting for a good opportunity to try it out. The opportunity came in the shape a the new project I am working on which requires a lucene grade search engine and is in lift/sacala which lends itself well as I could use the java elasticsearch client api.

Elasticsearch is quite new, and I could not find real documentation on how to integrate it, however, going over the java source code and high level documentation on the web site, I was able to come up with the following simple integration:

First define a search engine abstraction:

object SearchEngine extends Logger
{
 private var _node:Node = null
 private var _client:Client = null
 
 def startup()
 {
  if (!enabled)
  {
   warn("search engine is disabled. if this is not intentional, please change the configuration file accordingly")
   return
  }
  
  _node = nodeBuilder().client(true).node
  if (null == _node) return
  _client = _node.client
 }
 
 def shutdown()
 {
  if (null != _node) _node.close();
  _client = null
 }
 
 def enabled:Boolean = "true" == (Props.get("search.enabled") openOr "false")
 
 def connected:Boolean = null != _client
 
 def index(indexName:String, typeName:String, identifier:String, json:JValue)
 {
  if (null == indexName) throw new Exception("invalid (null) index")
  if (null == typeName) throw new Exception("invalid (null) type")
  if (null == identifier) throw new Exception("invalid (null) identifier")
  if (null == json) throw new Exception("invalid (null) json")
  
  if (!enabled) return
  confirmConnection
  
  val request:IndexRequestBuilder = _client.prepareIndex(indexName, typeName, identifier)
  request.setSource(pretty(render(json)))
  val response:IndexResponse = request.execute().actionGet() 
 }
 
 def delete(indexName:String, typeName:String, identifier:String)
 {
  if (null == indexName) throw new Exception("invalid (null) index")
  if (null == typeName) throw new Exception("invalid (null) type")
  if (null == identifier) throw new Exception("invalid (null) identifier")
  
  if (!enabled) return
  confirmConnection
  
  val request:DeleteRequestBuilder = _client.prepareDelete(indexName, typeName, identifier)
  val response:DeleteResponse = request.execute().actionGet()
 }
 
 def search(indexName:String, query:XContentQueryBuilder, from:Integer, size:Integer, explain:Boolean=false):SearchHits =
 {
  if (null == indexName) throw new Exception("invalid (null) index")
  if (null == query) throw new Exception("invalid (null) query")
 
  if (!enabled) return null
  confirmConnection
 
  val request:SearchRequestBuilder = _client.prepareSearch(indexName)
  request.setSearchType(SearchType.QUERY_THEN_FETCH) 
  request.setQuery(query) 
  request.setFrom(from.intValue) 
  request.setSize(size.intValue) 
  request.setExplain(explain) 
 
  val response:SearchResponse = request.execute().actionGet()
  return response.hits
}
 
 private def confirmConnection
 {
  if (!connected) startup
  if (!connected) throw new Exception("cannot connect to search engine, perhaps it needs to be disabled?")
 }
 
}

Naturally, the next step is to add a call into "SearchEngine.startup" from your Boot.scala so the app connects to the search server on startup.

Next define a pair of model & companion traits to hide the integration details from the domain objects:

trait SearchableModelMeta[T <: SearchableModel[T]] extends BaseModelMeta[T]
{
 self: T  with SearchableModelMeta[T] with BaseModelMeta[T] =>
 
 private def searchIndexName:String = this.pluralXmlName
 
 private def searchTypeName:String = this.xmlName
 
 override def beforeSave = updateSearchIndexDate _ :: super.beforeSave

 override def afterSave = storeInSearchIndex _ :: super.afterSave
 
 override def afterDelete = deleteFromSearchIndex _ :: super.afterSave
 
 def reindexAll()
 {
  findAll.foreach((instance:T) => { storeInSearchIndex(instance) })
 }
 
 def search(query:XContentQueryBuilder, from:Integer=0, size:Integer=100):SearchHits =
 {
  SearchEngine.search(this.searchIndexName, query, from, size)
 }
 
 private def updateSearchIndexDate(instance:T)
 {
  instance.indexedAt(Helpers.now)
 }

 private def storeInSearchIndex(instance:T)
 {
  SearchEngine.index(this.searchIndexName, this.searchTypeName, instance.id.toString, instance.toJson("search"))
 }
 
 private def deleteFromSearchIndex(instance:T)
 {
  SearchEngine.delete(this.searchIndexName, this.searchTypeName, instance.id.toString)
 }
}

trait SearchableModel[T <: BaseModel[T]] extends BaseModel[T]
{
 self: T =>
 
 /*
 *** columns
 */
 
 object indexedAt extends MappedDateTime(this.asInstanceOf[T])
 {
  override def dbColumnName = "indexed_at"
 }
}

Note this example uses a custom base model class, which is not super important to what I am trying to show here (it facilitates dynamic json assembly and naming conventions, but those are pretty straight forward). however, it is important to note that that base model mixes in LongKeyedMapper and IdPk. headers for the class & companion below.

trait BaseModelMeta[T <: BaseModel[T]] extends LongKeyedMetaMapper[T]

trait BaseModel[T <: LongKeyedMapper[T]] extends LongKeyedMapper[T] with IdPK

Last mixin the SearchableMode traits with the domain model so you can do things like:

Employee.search(...)

This of course if a very basic example which uses a handful of the available features elasticsearch and lucene offer, however, this should give anyone trying to integrate elasticsearch with lift a good head start.

ruby on rails style routes with lift

2010-11-04T17:00:00.000-07:00

In my latest project i am using lift/scala for restful services. spending the last couple of years using ruby on rails for such, the first thing i was looking for was a convenient way to define my generic restful routes. here is what i cam up with:

object Routes extends RestHelper with Logger
{      
 // response builder
 implicit def cvt:JxCvtPF[ResponseItem] = 
 { 
  case (XmlSelect, response, request) => response.toXml 
  case (JsonSelect, response, request) => response.toJson
 }
 
 private val PREFIX:String = "api"
 
 private var _services:mutable.ListMap[String, BaseService] = null
    
 def registerService(service:BaseService, alias:String=null)
 {
  if (null == _services) _services = new mutable.ListMap[String, BaseService]()
  val key:String = if (null != alias) alias else service.getClass.getName.split("\\.").toList.last.replace("Service$", "")
  _services += key -> service
 }  
   
 /* 
 *** standard restful routes
 */  
   
 serveJx
 {
  // GET /api/service/index.{xml|json}
  case Get(PREFIX :: StringValue(service) :: "index" :: Nil, _) => invokeApi(service, "index")   
 }
 serveJx
 { 
  // GET /api/service/1.{xml|json}
  case Get(PREFIX :: StringValue(service) :: LongValue(id) :: Nil, _) => invokeApi(service, "get", id)
 }
 serveJx
 { 
  // GET /api/service/api.{xml|json}
  case Get(PREFIX :: StringValue(service) :: StringValue(api) :: Nil, _) => invokeApi(service, api)
 }
 serveJx
 { 
  // GET /api/service/api/1.{xml|json}
  case Get(PREFIX :: StringValue(service) :: StringValue(api) :: LongValue(id) :: Nil, _) => invokeApi(service, api, id)
 }  
 serveJx
 { 
  // POST /api/service.{xml|json}
  case Post(PREFIX :: StringValue(service) :: Nil, _) => invokeApi(service, "create")
 }
 serveJx
 { 
  // POST /api/service/api.{xml|json}
  case Post(PREFIX :: StringValue(service) :: StringValue(api) :: Nil, _) => invokeApi(service, api)
 }
 serveJx
 { 
  // POST /api/service/api/1.{xml|json}
  case Post(PREFIX :: StringValue(service) :: StringValue(api) :: LongValue(id) :: Nil, _) => invokeApi(service, api, id)
 }  
 serveJx
 { 
  // PUT /api/service/1.{xml|json}
  case Put(PREFIX :: StringValue(service) :: LongValue(id) :: Nil, _) => invokeApi(service, "update", id)
 }
 serveJx
 { 
  // DELETE /api/service/1.{xml|json}
  case Delete(PREFIX :: StringValue(service) :: LongValue(id) :: Nil, _) => invokeApi(service, "delete", id)
 }
 
 private def invokeApi(service:String, api:String, id:Long=0):Box[ResponseItem] =
 {
  val serviceName:String = StringHelpers.camelify(service)
  val methodName:String = StringHelpers.camelifyMethod(api)
  
  info("processing api " + service + ":" + api + " as " + serviceName + "[Service]:" + methodName)
     
  if (!_services.contains(serviceName)) return Full(Failed("unknown service " + service))
  
  _services(serviceName) match
  {
   case serviceHandler:BaseService =>
   {     
    serviceHandler.getClass.getMethods.foreach((method:Method) =>
    {      
     if (methodName == method.getName)
     {      
      val result:Object = if (0 == id) method.invoke(serviceHandler) else method.invoke(serviceHandler, id.asInstanceOf[AnyRef])       
      result match
      {
       case response:ResponseItem => return Full(response) 
       case _ => return Empty
      }
     }
    })
    return Full(Failed("unknown API " + service + ":" + api)) 
   }
   case _ => Full(Failed("invalid API configuration, service is not a BaseService"))
  }
 }
}

Once the routes handler (above) is define, you need to register the service implementation(s) to the routes handler and register the routes handler to Lift's dispatch table, this is done in lift's boot sequence, typically defined in Boot.scala. for example:

Routes.registerService(SessionService)
Routes.registerService(UserService)
.    
.
.

LiftRules.dispatch.append(Routes)

Naturally, you can extend this rest handler to handle custom restful routes, using the serveJx or serve directives, this is explained in more details here

Please note that a bug in scala is preventing from bundling all those serveJx case statements together, thus, the DRYlessness.

comparing ruby 1.9 performance on ec2

2009-12-08T16:01:00.000-08:00

benchmarked using ruby's benchmarking suite. all machine are brand new fedora 8 basic instances, not running any additional services. all tests done on ruby 1.9.1p376 (2009-12-07 revision 26041) [i686-linux] compiled from source with --enable-shared flag. benchmarks performed twice on 2 separate instances.

results speak for themselves, don't forget to compare ec2 pricing ;)


	small 32 bit			medium 32 bit			large 64 bit
test	run 1	run 2	avg	run 1	run 2	avg	run 1	run 2	avg
app_answer	0.251	0.263	0.257	0.113	0.118	0.1155	0.087	0.111	0.099
app_erb	2.926	2.911	2.9185	1.2	1.24	1.22	1.178	1.184	1.181
app_factorial	1.425	1.435	1.43	0.573	0.574	0.5735	0.445	0.459	0.452
app_fib	2.935	3.071	3.003	1.248	1.291	1.2695	1.137	1.158	1.1475
app_mandelbrot	1.343	1.348	1.3455	0.566	0.569	0.5675	0.545	0.561	0.553
app_pentomino	99.62	99.227	99.4235	41.587	41.832	41.7095	38.587	38.976	38.7815
app_raise	3.155	3.087	3.121	1.309	1.326	1.3175	1.488	1.521	1.5045
app_strconcat	2.714	2.754	2.734	1.173	1.189	1.181	1.001	1.07	1.0355
app_tak	4.161	4.174	4.1675	1.743	1.756	1.7495	1.52	1.57	1.545
app_tarai	3.194	3.164	3.179	1.364	1.392	1.378	1.23	1.281	1.2555
app_uri	5.652	5.657	5.6545	2.372	2.381	2.3765	2.456	2.412	2.434
io_file_create	1.345	1.351	1.348	0.579	0.569	0.574	0.872	0.891	0.8815
io_file_read	1.379	1.372	1.3755	0.652	0.678	0.665	0.757	0.737	0.747
io_file_write	1.086	1.124	1.105	0.503	0.514	0.5085	0.394	0.413	0.4035
loop_for	8.137	8.393	8.265	3.485	3.685	3.585	2.924	2.924	2.924
loop_generator	2.993	3.061	3.027	1.301	1.305	1.303	1.274	1.278	1.276
loop_times	7.592	6.963	7.2775	3.423	2.857	3.14	2.614	2.657	2.6355
loop_whileloop	3.335	3.449	3.392	1.463	1.476	1.4695	1.067	1.059	1.063
loop_whileloop2	0.7	0.725	0.7125	0.33	0.327	0.3285	0.23	0.227	0.2285
so_ackermann	3.488	3.431	3.4595	1.459	1.479	1.469	1.321	1.336	1.3285
so_array	8.03	8.002	8.016	3.445	3.477	3.461	3.015	3.077	3.046
so_binary_trees	2.21	2.157	2.1835	0.892	0.905	0.8985	0.853	0.864	0.8585
so_concatenate	2.426	2.334	2.38	1.004	1.026	1.015	0.862	0.86	0.861
so_count_words	1.534	1.583	1.5585	0.682	0.698	0.69	0.631	0.628	0.6295
so_exception	6.177	5.951	6.064	2.483	2.473	2.478	2.754	2.712	2.733
so_fannkuch	122.375	122.345	122.36	51.535	51.816	51.6755	54.612	55.241	54.9265
so_fasta	16.102	15.909	16.0055	6.739	6.653	6.696	6.712	6.567	6.6395
so_k_nucleotide	9.231	9.204	9.2175	4.021	3.972	3.9965	3.861	3.895	3.878
so_lists	1.851	1.996	1.9235	0.855	0.845	0.85	0.782	0.778	0.78
so_mandelbrot	49.539	49.33	49.4345	20.966	21.093	21.0295	19.441	19.679	19.56
so_matrix	2.18	2.234	2.207	0.945	0.978	0.9615	0.819	0.858	0.8385
so_meteor_contest	31.53	31.425	31.4775	13.338	13.479	13.4085	12.317	12.254	12.2855
so_nbody	43.097	42.141	42.619	18.029	18.12	18.0745	18.64	18.931	18.7855
so_nested_loop	6.564	6.656	6.61	2.668	2.796	2.732	2.219	2.275	2.247
so_nsieve	15.524	15.361	15.4425	6.54	6.718	6.629	6.526	6.544	6.535
so_nsieve_bits	20.059	20.106	20.0825	8.496	8.534	8.515	6.785	7.031	6.908
so_object	4.818	4.795	4.8065	2.066	2.068	2.067	1.844	1.825	1.8345
so_partial_sums	56.984	57.449	57.2165	24.446	24.164	24.305	24.598	24.707	24.6525
so_pidigits	9.494	9.554	9.524	3.944	3.925	3.9345	2.865	2.846	2.8555
so_random	2.318	2.349	2.3335	0.992	1.016	1.004	0.915	0.941	0.928
so_reverse_complement	26.634	31.414	29.024	11.043	11.656	11.3495	12.476	12.484	12.48
so_sieve	0.428	0.452	0.44	0.21	0.227	0.2185	0.135	0.169	0.152
so_spectralnorm	22.134	22.296	22.215	9.6	9.451	9.5255	8.536	8.462	8.499
vm1_block*	7.212	7.208	7.21	3.409	3.134	3.2715	2.769	2.819	2.794
vm1_const*	2.143	1.988	2.0655	1.296	0.872	1.084	0.661	0.672	0.6665
vm1_ensure*	0.385	0.368	0.3765	0.15	0.168	0.159	0.143	0.153	0.148
vm1_ivar*	7.196	7.12	7.158	2.989	3.004	2.9965	2.735	2.741	2.738
vm1_ivar_set*	7.063	6.819	6.941	2.803	2.974	2.8885	2.492	2.502	2.497
vm1_length*	3.93	3.694	3.812	1.638	1.678	1.658	1.124	1.159	1.1415
vm1_neq*	2.814	2.818	2.816	1.221	1.236	1.2285	0.887	0.888	0.8875
vm1_not*	1.531	1.526	1.5285	0.651	0.702	0.6765	0.519	0.547	0.533
vm1_rescue*	0.324	0.208	0.266	0.133	0.153	0.143	0.112	0.144	0.128
vm1_simplereturn*	4.372	5.532	4.952	1.929	2.352	2.1405	1.731	1.661	1.696
vm1_swap*	1.714	1.631	1.6725	0.73	0.724	0.727	0.484	0.522	0.503
vm2_array*	2.747	2.719	2.733	1.164	1.142	1.153	1.034	1.04	1.037
vm2_case*	0.591	0.6	0.5955	0.254	0.259	0.2565	0.256	0.275	0.2655
vm2_eval*	102.12	104.003	103.0615	42.756	42.905	42.8305	42.582	43.292	42.937
vm2_method*	6.839	7.739	7.289	2.926	2.914	2.92	2.515	2.531	2.523
vm2_mutex*	6.675	6.75	6.7125	2.827	2.857	2.842	2.435	2.381	2.408
vm2_poly_method*	10.517	10.217	10.367	4.385	4.578	4.4815	3.747	3.577	3.662
vm2_poly_method_ov*	0.905	0.821	0.863	0.357	0.408	0.3825	0.385	0.412	0.3985
vm2_proc*	2.923	2.917	2.92	1.31	1.219	1.2645	1.118	1.115	1.1165
vm2_regexp*	7.88	7.773	7.8265	3.438	3.377	3.4075	2.713	2.722	2.7175
vm2_send*	1.066	1.085	1.0755	0.463	0.463	0.463	0.427	0.428	0.4275
vm2_super*	1.866	1.87	1.868	0.788	0.869	0.8285	0.696	0.735	0.7155
vm2_unif1*	0.845	0.884	0.8645	0.41	0.415	0.4125	0.343	0.368	0.3555
vm2_zsuper*	1.937	2.042	1.9895	0.821	0.849	0.835	0.742	0.777	0.7595
vm3_gc	5.348	5.301	5.3245	2.234	2.251	2.2425	2.357	2.366	2.3615
vm3_thread_create_join	5.113	5.263	5.188	2.353	2.289	2.321	3.317	3.288	3.3025
vm3_thread_mutex	1.818	1.862	1.84	32.556	35.779	34.1675	4.661	6.225	5.443

rails on ruby 1.9.1 on ec2

2009-12-08T01:46:00.000-08:00

amazon ec2 currently offers fedora servers preinstalled with ruby 1.8.6 which is used by its ami tools. while this is a good basic configuration, one may want to upgrade to ruby 1.9.1 which offers many improvements, performance being the key one.

upgrading to ruby 1.9.1 on an ec2 fedora server, and running a rails application on top of it is a multi step process but a straight forward one. below are the steps I took to get my amazon ec2 hosted production environment migrated to ruby 1.9.1

step I: launch your ec2 instance
this post assumes you know how to launch an ec2 instance, in my case i used ec2's 64bit basic fedora ami, i found the 32bit unusable due to stability issues i experienced with the original kernel. more details on this are logged here

step II: install ruby 1.9.1
first we want to get the dependancies out of the way, in my case i needed to install several libraries that will be later used by my rails app, you may have a smaller or larger dependencies list depending on the gems you need to run.

yum update
yum install httpd httpd-devel mysql mysql-devel libxml2 libxml2-devel gcc-c++

now lets install ruby 1.9.1, I choose to install in in my /opt directory, but you may choose elsewhere. we will be suffixing ruby with the "19" suffix so that we can run ruby 1.9.1 side by side with the existing ec2's ruby 1.8.6

cd /opt
wget ftp://ftp.ruby-lang.org/pub/ruby/1.9/ruby-1.9.1-p376.tar.gz
tar xvfz ruby-1.9.1-p376.tar.gz
cd ruby-1.9.1-p376
./configure --enable-shared --program-suffix=191
make
make install

now that we have ruby 1.9 compiled and installed, it's time to do some renaming and symbolic linking so that the system uses ruby 1.9.1 by default while overcoming ec2's hard-wired ruby 1.8.6 installation

mv /usr/bin/ruby /usr/bin/ruby186
mv /usr/bin/irb /usr/bin/irb186
mv /usr/bin/erb /usr/bin/erb186
mv /usr/bin/testrb /usr/bin/testrb186
mv /usr/bin/gem /usr/bin/gem186
mv /usr/bin/rake /usr/bin/rake186
mv /usr/bin/ri /usr/bin/ri186
mv /usr/bin/rdoc /usr/bin/rdoc186

ln -s /usr/local/bin/ruby191 /usr/bin/ruby
ln -s /usr/local/bin/rake191 /usr/bin/rake  
ln -s /usr/local/bin/gem191 /usr/bin/gem  
ln -s /usr/local/bin/irb191 /usr/bin/irb  
ln -s /usr/local/bin/erb191 /usr/bin/erb  
ln -s /usr/local/bin/ri191 /usr/bin/ri  
ln -s /usr/local/bin/rdoc191 /usr/bin/rdoc  
ln -s /usr/local/bin/testrb191 /usr/bin/testrb

To confirm you are running the correct ruby version, run
>> ruby -v
you should see (assuming you download same revision as i did)
ruby 1.9.1p376 (2009-12-07 revision 26041) [i686-linux]

step III: installing rails
this step can prove to be more involved than others, depending on the gems you need to run. I have a detailed post on this here. the original post was written for OSX, but with the minor differences in ruby's location is perfectly correct any *nix.

step IV: bundling your custom AMI
now that we are done setting up the server, the next natural step is to bundle it as a custom ami so we can reuse it.

an issue we run into is that ec2's ruby based ami tools are not ruby 1.9 compatible, more specifically the ec2_upload_bundle tool which we need to use in order to bundle. luckily, we installed ruby 1.9.1 side by side to the original 1.8.6, so we have an easy way out: simply use ruby 1.8.6 to run the tools. to do so, change ec2 wrapper scripts in /usr/local to use ruby18 instead of ruby. for example change /usr/local/bin/ec2-bundle-vol from

#!/bin/bash
ruby /usr/lib/site_ruby/ec2/amitools/bundlevol.rb $*

#!/bin/bash
ruby186 /usr/lib/site_ruby/ec2/amitools/bundlevol.rb $*

apply the same principle to the rest of the ec2 wrapper scripts in /usr/local

Fixing soap4r for ruby 1.9

2009-10-07T13:05:00.000-07:00

soap4r 1.5.8 is broken on ruby 1.9, below are the steps I took to fix it. Please note I have not attempted to fix soap4r completely, and that this fix does not go beyond the purposes of my usage, there are likely to be areas that are still broken under 1.9.

lib/soap/baseData.rb

replace

TypeParseRegexp = Regexp.new('^(.+)\[([\d,]*)\]$', nil, 'NONE')

with

TypeParseRegexp = Regexp.new('^(.+)\[([\d,]*)\]$')

lib/soap/generator.rb

replace

Regexp.new("[#{EncodeMap.keys.join}]", nil, XSD::Charset.encoding)

with

Regexp.new("[#{EncodeMap.keys.join}]")

lib/soap/property.rb

replace

COMMENT_REGEXP = Regexp.new('^(?:#.*|)$', nil, 'u')
CATDEF_REGEXP = Regexp.new("^\\[\\s*#{KEY_REGSRC}\\s*\\]$", nil, 'u')
LINE_REGEXP = Regexp.new("^#{DEF_REGSRC}$", nil, 'u')

with

COMMENT_REGEXP = Regexp.new("^(?:#.*|)$")
CATDEF_REGEXP = Regexp.new("^\\[\\s*#{KEY_REGSRC}\\s*\\]$")
LINE_REGEXP = Regexp.new("^#{DEF_REGSRC}$")

remove / comment out

# for ruby/1.6.
unless Enumerable.instance_methods.include?('inject')
  module Enumerable
    def inject(init)
      result = init
      each do |item|
       result = yield(result, item)
      end
      result
    end
  end
end

lib/soap/rpc/element.rb

replace

type_qname = TypeMap.index(mapped_class)

with

type_qname = TypeMap.key(mapped_class)

lib/xsd/charset.rb

replace

@internal_encoding = $KCODE

with

@internal_encoding = "UTF8"

replace

USASCIIRegexp = Regexp.new("\\A#{us_ascii}*\\z", nil, 'NONE')

with

USASCIIRegexp = Regexp.new("\\A#{us_ascii}*\\z")

replace

EUCRegexp = Regexp.new("\\A#{character_euc}*\\z", nil, 'NONE')

with

EUCRegexp = Regexp.new("\\A#{character_euc}*\\z", nil, 'n')

replace

SJISRegexp = Regexp.new("\\A#{character_sjis}*\\z", nil, 'NONE')

with

SJISRegexp = Regexp.new("\\A#{character_sjis}*\\z", nil, 'n')

replace

character_utf8 =
"(?:#{us_ascii}|#{twobytes_utf8}|#{threebytes_utf8}|#{fourbytes_utf8})"
UTF8Regexp = Regexp.new("\\A#{character_utf8}*\\z", nil, 'NONE')

with

character_utf8 = "(?:#{us_ascii}|#{twobytes_utf8}|#{threebytes_utf8}|#{fourbytes_utf8})"
UTF8Regexp = Regexp.new("\\A#{character_utf8}*\\z", nil, 'n')

lib/xsd/ns.rb

replace

ParseRegexp = Regexp.new('\A([^:]+)(?::(.+))?\z', nil, 'NONE')

with

ParseRegexp = Regexp.new('\A([^:]+)(?::(.+))?\z')

lib/xsd/xmlparser.rb

replace

NSParseRegexp = Regexp.new('^xmlns:?(.*)$', nil, 'NONE')

with

NSParseRegexp = Regexp.new("^xmlns:?(.*)$")

replace

raise LoadError unless XSD::XMLParser.constants.find { |c| c.downcase == name }

with

raise LoadError unless XSD::XMLParser.constants.find { |c| c.to_s.downcase == name.downcase }

Running a Ruby on Rails Application on OSX with Ruby 1.9

2009-10-02T22:30:00.000-07:00

General

OSX comes preinstalled with ruby 1.87, however, ruby 1.9 has out there for quite a while now and does offer improvements in many areas, including performance, which is always a good trigger for an upgrade. A deeper overview can be found here.

Ruby 1.9

You can use MacPorts, Fink or compile from source. Hivelogic has a good post on how to install ruby on OSX here. I have chosen to use MacPorts as I use it for other ports:

$ sudo port install ruby1.9

If you are doing a side by side installation (i.e keeping you original ruby installation in place), you may want to do some symbolic linking to make sure the ruby, irb, gems, rake, ri and rdoc are pointing the 1.9 version. If you choose to skip this step, remember to use the ruby1.9, irb1.9, etc command instead.

Disclaimer: Please be careful with the below, the order and locations matter and one installation (yours) can be different than other (mine), if you are not familiar with OSX, bash and symbolic linking I suggest you do some readying before continuing with this step or just stick to using the 1.9 command namespace (See above).

The example below assume you ruby 1.9 is installed at /opt/local/bin/ruby1.9 (MacPort default location) and that your original ruby installation was a /System/Library/Frameworks/Ruby.framework/Versions/1.8 (OSX default location).

$ cd /System/Library/Frameworks/Ruby.framework/Versions
$ mkdir 1.9.1
$ mkdir 1.9.1/usr
$ mkdir 1.9.1/usr/bin

$ cd 1.9.1/usr/bin
$ ln -s /opt/local/bin/ruby1.9 ruby
$ ln -s /opt/local/bin/rake1.9 rake
$ ln -s /opt/local/bin/gem1.9 gem
$ ln -s /opt/local/bin/irb1.9 irb
$ ln -s /opt/local/bin/erb1.9 erb
$ ln -s /opt/local/bin/ri1.9 ri
$ ln -s /opt/local/bin/rdoc1.9 rdoc
$ ln -s /opt/local/bin/testrb1.9 testrb

$ cd /System/Library/Frameworks/Ruby.framework/Versions
$ rm Current
$ ln -s 1.9.1 Current

$ cd /usr/bin
#for some reason apple is linking directly to the v. 1.8 folder instead of the current directory, we will "fix" that.
$ mv gem gem1.8
$ ln -s /System/Library/Frameworks/Ruby.framework/Versions/Current/usr/bin/gem
#for some reason apple has put the rake executable in /usr/bin instead of in the current; directory, we will "fix" that.
$ mv rake rake1.8
$ ln -s /System/Library/Frameworks/Ruby.framework/Versions/Current/usr/bin/rake

To confirm you are running the correct ruby version:
$ ruby -v
$ gem env
$ irb
>> RUBY_VERSION

Rails

As you are installing a new ruby, you also need to reinstall all of you gems, as this artical is focused on ruby on rails, we will start with the rails gem. There is nothing exciting here, simply run the standard rails installation command

$ gem update --system
$ gem install rails

Other Gems

For the purposes of my application, I have installed the follwing gems:
actionmailer (2.3.4, 2.3.3)
actionpack (2.3.4, 2.3.3)
activerecord (2.3.4, 2.3.3)
activeresource (2.3.4, 2.3.3)
activesupport (2.3.4, 2.3.3)
builder (2.1.2)
cgi_multipart_eof_fix (2.5.0)
daemons (1.0.10)
fast_xs (0.7.3)
fastthread (1.0.7)
gem_plugin (0.2.3)
httpclient (2.1.5.2)
libxml-ruby (1.1.3)
macaddr (1.0.0)
mongrel (1.1.6)
mongrel_cluster (1.0.5)
mysql (2.8.1)
rack (1.0.0)
rails (2.3.4, 2.3.3)
rake (0.8.7)
rubygems-update (1.3.5)
rubyzip (0.9.1)
soap4r (1.5.8)
uuid (2.0.2)

Hacks

This is where the fun begins, some of the gems and rails modules are not fully baked for ruby 1.9, below are a list of hacks that I had to plug in to make my rails app work. Naturally, your case may be different, a good reference to gems under 1.9 can be found here.

mongrel gem
The 1.1.5 mongrel gem does not install over ruby 1.9. You may choose to fix it manually or install 1.1.6 from gems.rubyinstaller.org:

$ gem install mongrel --source http://gems.rubyinstaller.org

More info on this can be found here.

mongrel cluster
Needs a small syntax patch, fix the case statement at the end of the mongrel_cluster_ctl by replacing ":" with "then".

restful_authentication plugin
If you are using restful_authentication plugin you are likely to crash as soon as the plugin attempts to read data from cookies. This is due to a bug in active_support module. There is a documented patch here.

uploading
Most applications use some sort of uploading, mine does too. Rack 1.0.0 on 1.9 crashes does to unicode issues. There is a documented patch here.

ftools
ftools has been deprecated in 1.9, if your code is dependent on it, move to fileutils.

rubyzip gem
Uses ftools, you will need to remove the dependency. There is a documented patch here.

libxml-ruby gem
I found this gem to be unstable on 1.9, throwing many memory allocation errors. The solution was to update the libxml libs and then install the gem against the updated sources:

$ port install libxml2
$ gem install libxml-ruby -- --with-xml2-include=/opt/local/include/libxml2

More info on this can be found here.

soap4r
Broken, I monkey patched locally which was enough for my usage, but not a compelte solution. More info here.

These steps got my app on its feet, however, I am still testing stability and performance under the new setup. I will update this post if and when new problems or solutions come up.

Compiling mod_xsendfile on OSX

2009-10-01T19:55:00.000-07:00

mod_xsendfile is an apache2 module which come handy when you need to serve static files through a secondary application layer or a proxied web server, both tend to serve static files slower and less efficiently than apache does. In such cases you want to delegate the request (file path) to apache so that the secondary layer/proxied web server are freed to move on and process other requests. A common example is a php download handler that needs to go to the db to fetch the file path, check permissions, etc but don't want to deal with the actual file transfer, another common example is a ruby on rails proxied web server such as mongrel which is slow in serving static files under stress due it's concurrency model and can quickly become a bottle neck. For more information on mod_xsendfile and how to configure it go here.

The above information well known, however mod_xsendfile installation documentation states the following:

1. Grab the source.
2. Compile and install:
>> apxs -cia mod_xsendfile.c
3. Restart apache
4. That's all.

If you try this on OSX the compilation step will fail due to missing architecture flags, hence, use the following instead:

>> apxs -cia -Wc,"-arch x86_64 -arch ppc -arch i386 -arch ppc64" -Wl,"-arch x86_64 -arch ppc -arch i386 -arch ppc64" mod_xsendfile.c

This will compile mod_xsendfile in all 4 OSX flavors, which should cover your case. Naturally, you can change the architecture flag to just the one that is relevant to your hardware. More info here.

Configuring CruiseControl as an OSX Startup Daemon

2009-09-30T19:31:00.000-07:00

General

The purpose of this post is to describe how to configure a startup daemon for cruisecontrol on OSX. While CruiseControl ships with a startup script (cruisecontrol.sh), one would like CruiseControl to automatically start after system startup to avoid the manual step of starting it.

This post assumes previous knowledge with OSX, bash, and CruiseControl and does not attempt to cover how to install or generally configure CruiseControl on OSX. There are several articles covering the latter, while the topic of configuring CruiseControl startup daemon is only partially covered.

Creating the Daemon

• Navigate to the startup daemons folder

$ cd /Library/LaunchDaemons

• Create a new plist file named org.sourceforge.cruisecontrol.plist, the plist file will look something along these lines:

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple Computer//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
 <dict>
         <key>Label</key>
         <string>org.sourceforge.cruisecontrol</string>
         <key>WorkingDirectory</key>
         <string>/usr/local/cruisecontrol</string>
         <key>ProgramArguments</key>
         <array>
                 <string>/System/Library/Frameworks/JavaVM.framework/Home/bin/java</string>
                 <string>-Xms128m</string>
                 <string>-Xmx256m</string>
                 <string>-Djavax.management.builder.initial=mx4j.server.MX4JMBeanServerBuilder</string>
                 <string>-Dcc.library.dir=/usr/local/cruisecontrol/lib</string>
                 <string>-Djetty.logs=/usr/local/cruisecontrol/logs</string>
                 <string>-jar</string>
                 <string>/usr/local/cruisecontrol/lib/cruisecontrol-launcher.jar</string>
                 <string>-jmxport</string>
                 <string>8000</string>
                 <string>-webport</string>
                 <string>8080</string>
                 <string>-rmiport</string>
                 <string>1099</string>
         </array>
         <key>EnvironmentVariables</key>
         <dict>
         </dict>
         <key>RunAtLoad</key>
         <true/>
         <key>Uses</key>
         <array>
                 <string>Network</string>
                 <string>Resolver</string>
         </array>
 </dict>
</plist>

Whats in the plist file?

• Label => Name of daemon, set to org.sourceforge.cruisecontrol

• WorkingDirectory => Location of the CruiseControl installation

• ProgramArguments => Java calls to cruisecontrol-launcher jar including configuration for CruiseControl port numbers, libs and log location. Add additional parameters as you see fit

• RunAtLoad => Run at startup, set to true

• Uses => Dependency on other services, in our case networking and resolver

Testing the Daemon

• Loading:

$ launchctl load org.sourceforge.cruisecontrol.plist

Use the above command to load the daemon. The system will do the same upon reboot, so this is only required when you are testing the daemon.

• Unloading:

$ launchctl unload org.sourceforge.cruisecontrol.plist

Use the above command to unload the daemon if and when it fails and you need to reload it again.

• Testing to see that the daemon is actually loaded

$ launchctl list

If everything went well, you should see you daemon listed (org.sourceforge.cruisecontrol) with a valid pid.

You can also monitor the startup process on cruiscontrol.log, or if you access to he UI, through the system's console.