Java 1.8 Setup for R and R Studio

Why is this so hard! There are a lot of posts out there all trying to help people get java packages working with R Studio and R on MacOSX (El Capitan, 10.11.5 in my case). I also have Java 8 installed and would prefer to use that.

This post, seemed to help me understand best what the issues were but alas, didn’t get me running the packages I wanted: FSelector which depends on RWeka and rJava. The R config file, /Library/Frameworks/R.framework/Versions/3.3/Resources/etc/Renvironis an important key, but setting variables there did not help in the way that post suggested. Here is what I did.

With a clean Renviron file, and both jdk 1.6 (RStudio seems to require this to do anything with java) from apple, and jdk1.8 from Oracle installed, jdk 8 as the default, I ran

sudo R CMD javareconf

if successful you will see a compiled program success in the output. Unfortunately you aren’t done yet. Start R and use

install.packages(‘rJava’, type=’source)

It was downloaded, compiled and ran a test compilation for me, and this worked for me but there are a couple of steps needed before a full package can actually be executed. First I needed to install X, from This will install X libraries under /opt/X11 that java looks for at run time. If you were having problems running the test program in the previous step, this may solve it for you.

Finally, you do need to link the jvm shared dylib into a place the R can find it.

ln -f -s $(/usr/libexec/java_home)/jre/lib/server/libjvm.dylib /usr//locallib

and finally, we actually do need to edit the Renviron file mentioned earlier to tell the dyn loader where to find the library. Edit the end of the file so it looks like this (/Library/Frameworks/R.framework/Versions/3.3/Resources/etc/Renviron):

Local Variables: ***
### mode: sh ***
### sh-indentation: 2 ***
### End: ***
## CRAN R ships with qpdf inside so use it unless directed otherwise

After that I was able to open R Studio and load and start FSelector



gains <- information.gain(someform~., somedata.table)

I hope this helps somebody!

Introduction to Docker talk and follow up

My slides and the code I used for the Introduction to Docker talk most recently done at are available in my github profile at

Thanks to all who came! One of the most interesting parts of the talk, helped by the smart people in the room who asked great questions, were the questions about when it is appropriate to use Docker or not. Docker is most often talked about in relationship to microservices, the architectural pattern where you split your application or system into many small, independent services, For this it is easy to say Docker is a great match. Lots of small applications running on a cluster of docker hosts is what it was built for. Another really useful thing is to use docker as an alternative to installing tons of software on your development box, or even on your servers. The files in a Docker image are completely isolated from the host and can be cleanly added and removed at will. You don’t have to create a linux, windows, macox, other versions of your software if people expect to use it via docker.

In the “Docker all the things” camp, you would hear the call to move everything this way, something you hear anyway when a new technology is early in the hype curve. I can say that in my experience so far, there have been some real wins running things in containers:

  • small myql instances used for small apps
  • redis running locally for development
  • riak running locally was much easier to test in docker

On the other hand, while it may have been easier to run jenkins locally in docker, running jenkins in docker at AWS has not been a clear winner: a dedicated host may have made more sense, perhaps setup with vagrant and bash or even ansible (which I now know enough about to say that, thanks Mike Putnam!). The image is over 1GB in size, and it consumes all the memory and cpu the host has available to run Jenkins and one agent. If we could get the image size down, running multiple agents in Docker might make sense though. Life is trade offs, no?

This is really the next phase of my longer term evaluation of Docker, to understand and to be able to more clearly express when it’s great, good and terrible to use Docker.

Solved! reason: File not found when installing Android SDK and NDK

Because this took such a ridiculous amount of time to solve, and because so many people seemed to have encountered it, I figured it would be good to document the error I got, the steps I took to resolve it, and finally what the solution was. First of all the error itself during the install:

Failed to fetch URL,
reason: File not found

Failed to fetch URL,
reason: File not found

If you search for these or variants of these errors with Android search terms you will find a LOT of questions and many similar answers. None of them worked for me, thought I could see how they would be a problem. In my case I was installing on a Windows VM running VMWare Fusion on a Macbook. Here is the log of what I tried, based on suggestions from a variety of sources, StackOverflow high among them:

Ran as administrator
verified no proxy setup
verified can run on this same computer’s mac osx install and load (confirmed no external firewall issues since this is working
tried the check https box
set permissions to folder as myself
set temp folder to folder on drive instead of \\vmware-host\….
set ndk folder to folder on drive instead of \\vmware-host\…

opened windows firewall to c:\windows\system32\java.exe
opened windows firewall to c:\program files (86)\Java\jre7\bin\java.exe
and the same for 64 bit under \program files\
set java_home to 32 bit version of java
added java runtime flags “”

tried bridged mode instead of NAT networking in vm-ware
tried the with my vpn on and off
tried rebooting just the vm
tried rebooting the mac

None of these things made any difference at all. However the early steps about the location of the temp folder and ndk folder were the best clue. The real issue is that java HAAATES spaces in file names and can’t deal with UNC paths, even those in this case mapped with a drive letter.

The answer was….
set the environment variable ANDROID_SDK_HOME to a location on a local drive instead of a UNC path.

Twin Cities Code Camp: Evolving REST to Hypermedia designs

In advance of my talk here is the sample code and slides for Twin Cities Code Camp 2014. The premise of the talk is the move to having web api’s that are primarily data oriented to those those that include actions appropriate for the context and time via links. Using HAL, it is possible, but you also need a state machine to “calculate” the appropriate links. For that I am releasing my pet side project to accomplish that as part of this talk.

The demo source is available at github. There is a branch of the framework in that repo in a zip file. I will maintain the main branch also at github, restructuring it now that the talk is over and I’m not wrasslin’ with nuget and the directory structure I’d like to use.

Note that the version of WebApi.HAL i’m using is not the current version. I had to add support for a few key HAL components, curies, and type and profile attributes, to accomplish what I needed. These have been submitted as pull requests, but they are still working through some previous pull requests so haven’t incorporated them as of yet. Use my fork if you want the source. For now I used a nuget package included in the zip. You will have to add a package source source to use it.

Basic Idea Economics

[note: I have been sitting on this post for a long time. I did take the opportunity to comment on software patents but still sat on it. Time to just let it go and take my beatings]

While traveling in Scotland a few years back, I had a sense of history like nowhere I’ve ever been before. It had a lot to do with the its mixture of medieval castles, grass, and sheep, dotted with a few little towns. It just had this feeling to it that many parts of life in Scotland had not changed for hundreds of years. For some reason this got me thinking about the evolution of what we think of capital and it’s new incarnation, intellectual capital. I also had the pleasure one of my many nights learning about single malt scotch to discuss this with an English industrialist. While I have may have glossed over many important details, I think my basic premise is sound. Capital is changing from buying shares in factories and other producers to buying ideas.


I see it something like this. In feudal times, wealth was based on owning property. Property was the only capital worth anything. If you had property, the property supported agriculture, like crops and the sheep I saw every day. Agriculture produces produce, and the produce was split between the tenants and the owners. The owners sell their portion for income. As a landowner, the more land you owned the  more tenants you could have living on the that land. This led to more production and thus more product you kept for yourself. Grass grew, it was processed by animals, the animals are processed by humans and viola, produce, sales and income. There was one key idea, the production of goods in the traditional way producing the traditional results. The intellectual capital in this was minimal and traditional. I would guess that the very idea would have seemed strange to people of the time.

In the industrial age, things did change. Capital was used to own the means of production in the form of factories. While prestige may have still been associated with ownership of property, the real bang for your buck was in amassing enough capital to build a factory. The factory typically organized a combination of labor and machines into the specialization needed to mass produce a small variety of products. After the factory was built the task of the manufacturer is to try to preserve existing markets, tune the process and equipment where possible to reduce ongoing costs to maximize the investment. This is mostly because of the large expense in building the facility and training the workforce. In this era, the number of working ideas went up quite a lot, but was still somewhat small. You needed the starting idea of the factory, the skills needed to run the equipment, and then after that,  just like in the days of sheep, you would try to preserve your environment and get away with as few changes as possible because change was expensive. Intellectual capital was alive however. Patents got their start, mostly as a way to encourage the large investment in factories by protecting them from competition for awhile, a little monopoly.

The age we are in now, probably dating back to the 1950’s, has had another major shift. I don’t think many entrepreneurs think about building a factory and making it produce for 50 years. Ownership of production is still important of course, but capitalists now own many shares of many companies, all representing many different products, ideas, markets and companies. More and more companies are valued on the ideas and people they represent. The term intellectual capital has become widely used, if not widely understood.

As a person that got his start in the software field working on open source software, the traditional meaning of the term, patents on ideas enforceable by law, has always seemed pretty repugnant to me. The idea is to exercise control much the same way a factory would exercise control over it’s physical assets: don’t use my ideas without my permission. Times have changed. Raw ideas are cheap to get and cheap to copy, and the enforcement effort is rarely worth the expense. Many ideas are encoded into software but software is better served by copyright than patents. In software most of the actual bits of a program are the shared libraries, device drivers, hardware and operating systems of the underlying general computer system. There are absolutely innovative ideas in software, but these are built on the shoulders of other’s works in smaller and smaller increments. The real intellectual capital exists in the heads of people who hold, nurture and grow the ideas and is inherently not patentable.

As I wrote in my letter to the USPTO, I think rather than focus so much on the specifics of how to judge whether an idea represented in software is patentable, we should go back to the reasons the patent was created in the first place. It was to create incentive for investment. Many intelligent people will argue that today that investment is in intellectual capital and it should be protected in the same way as the machinery of a factory was protected in the past. I think the fact is that the patent was to allow for the investment in the machinery that was very expensive and deserved the short term monopoly to patent provided. Today however we are better off focusing on faster delivery of more competing ideas without restrictions. The rapid growth of the internet is a wonderful example of this approach in practice and no one can argue against its overall economic benefits.

Why aren’t more people listening to jazz?

Many who know me have heard this before, but the topic keeps coming up, and really is important so I wanted to write this down so I can refer people to a more complete train of my own thoughts on this.

Stats vary but it generally believed that the size of the jazz audience is shrinking. Like anything that seems to be going in the wrong direction there are lots and lots of opinions about why that may be. This well written article that came across my Facebook feed today is one of may. Common themes are:

  • The music is too complex for average listeners
  • People are stupid/lazy/sheep like and can’t handle an art form like jazz
  • Music is now about the video, without video it’s just a bunch of noise
  • Without words, the music is just a bunch of noise
  • The “real jazz” stopped in the 60’s and went downhill into a morass of technicians with no soul
  • and so on


Nearly everybody that has a solution, and most just complain without suggesting anything, will bring up education as the way forward. I think this is wrong. How well has it worked out for classical music? There does have to be a place for people to get exposed to the music, but in school, as some sort of requirement? While sitting quietly in class? With tests? With some old guy or gal talking for an hour at a time about it? Without dancing?

That well written article does come close to what I believe helps us understand the problem. It talks about associations people have with words that make the music less abstract. It is a unique point of view, and there is no doubt some truth to it. Associations are what connects people to any art form, but music more so than others. Imagine the movie Jaws without the music. People know that music without the movie in a couple of notes. There is an emotional connection between the two. What would any movie be without the music? But it is not just movies. What was the music at some great party you were at? Were the people interesting to you, fun, made you feel great? You will be connected to that music in a way that never would have happened in a classroom. What music was playing on some wonderful evening with your girlfriend, spouse, partner? Were you moved? That music will always be a part of that experience. I think the reason pop music is so connected with people is just that it is what is playing where people are and as a result those people are bonded to that music.

Once you have that bond, you are hooked for life. Without that bond, it is just a bunch of noise

What we need is to find ways to include jazz in situations that also have some other social or emotional context to help people make that bond. It could be school, as anybody who struggles through a 3 month play or musical effort with late nights and before and after parties can attest too, but it doesn’t have to be in school and is not a classroom. These days, the clubs generally are non-smoking and serve food in addition to alcohol. Young people can go to clubs with the right planning and consideration. There needs  to be a scene, a scene where people are having fun with each other and in which the music is a central part. A scene fun for more reasons that just sitting quietly listening to the music and somehow abstractly contemplating it’s inner merits. That my friends is for people who are already fans.

I will be a Wells Fargo customer for life

Say what?

Banks in general have not gotten much good press these last few years, heck not for this last decade. I have shared the feeling that many have had that the newly unregulated banks had a big part to play in our financial crisis, though I think they were not the only parties responsible by a long shot. I’m not prone to conspiracy theories or deep class suspicion though, and view banks as a necessary service. I don’t love or hate them, just a commodity type of service one uses to handle money. So why would I choose to stay with one bank for life?

In order to go to a really great school, Oberlin College, my daughter Claire used a variety of funding sources: academic grants, savings, federal and private student loans. Her dual degrees in Creative Writing and Computer Science were very likely to make her employable and able to pay back her share of the loans. Like everybody else, we don’t understand why college tuition is going up faster than the rest of the economy, or why private student loan interest rates are so high. When many parts of the developed world give college education for free, how will we compete as students are choosing not to go at all, or leave school with enough dept to severely limit their economic mobility or to risk taking on new ventures or joining innovative start-ups? Well, regardless of all the negative aspects of borrowing for college, Claire like many, probably most of her generation, borrowed some money for college. Some if it from Wells Fargo.

College costs have been a big political football too, but not one many in congress have been willing to catch and run with. In 201o, there was some movement going on in congress to provided relief for student debt in the worst of cases: after the death or disability of the student. In these cases the co-signer of the loans, generally the parents who may not have really expected to fully bear a burden like this, becomes responsible, and with less rights than a regular borrower. Here is a link to more information about this. The proposed laws so far have not passed through congress. If you are a parent or family member helping a student out with these enormous debts by co-signing a loan, you had better think really hard about some sort of life insurance or you may find yourself in a terrible position should the worst happen to your child.

Wells Fargo decided to do something without a congressional mandate. At that time in 2010, though they said it was not related to the bills going through congress at the time, Wells Fargo quietly announced they would forgive loans in the event of death or permanent disability. They weren’t forced to do this by congress, nor did most of their competitors make a similar policy shift. I knew nothing about it at the time, nor would I have likely paid much attention if had known. As far as I was concerned, my daughter was doing great at school and would be paying back her student loans along with us as we had planned.

And then, in May 2013, she died unexpectedly, one year into a phd program at USC.

This unimaginable turn of events was the most wrenching, grueling, rip away your faith in all good things a parent could ever experience.  But wait, there’s more. If you are not careful, as I wasn’t, you are also now responsible for all of your child’s private student loan debt. Federal loans are already forgiven, but most private loans fall to the co-signer. There is no built in insurance policy like you might have gotten on a car loan.

This did not happen to us, thanks to Wells Fargo. There were two phone calls, an exchange of  legal documents, a condolence letter, and it was over. We still had our grief and our memories. But we were not also looking at a decade of debt.

So there it is. I’ve been a customer for over a decade anyway, had ups and downs, but have been generally satisfied the last few years. After this experience, I think I’ll stick around a lot longer. If you are a parent of a student with student loan debt, I’d recommend you pay careful attention to what could happen if the unimaginable happens to you.

Visual Studio + TFS error TF400030 workaround

I have been fighting with an error in Visual Studio I thought would be worth sharing as it seems many are fighting the very same issue. What happens is that if you have Visual Studio 2012, a reasonably large code base and local workspaces setup with TFS, the default, you may run into constant error pop ups with the message:

TF400030: The local data store is currently in use by another operation. Please wait and then try your operation again. If this error persists, restart the application.

According to this post,  Microsoft believes the problem is resolved. It seems worse with Update 2 though and the message states that their refactoring of this code will be available in the next release, i.e. post Update 2. So for now, you may be kind of stuck.

[update, 6/1/2013: MS has responded, but not come up with any good solutions here. Putting branches in different workspaces sounds promising though. Also I think based on the IO involved in how it’s implemented, an SSD is a must]

The workaround I found is pretty simple. Only have the Source Control Explorer open in one or less of your open instances of Visual Studio.

[update: today I had to close it in all my visual studio instances]

Another person on my team solved it a different way. He started removing extensions, and believes it was the Web Essentials extension removal that got him functional again. 

Hope it helps!

Working Remotely

A bit of conversation has been going on around the web first because of Yahoo’s decision to axe telecommuting, and today because of a nicely written article from Jeff Robbins, CEO of Lullabot, the Drupal company. Bob McWhirter who I worked with a long time ago on JDOM, had a very similar spin on this back in 2010 where he talked about how Twine changed from being a distributed team to a co-located team leaving him as a remote worker.

I have worked a lot with off shore remote workers and I am currently working remotely most days of the week. This really can work, but as these articles point out, you must do this very consciously, taking time to think about your communication patterns that (sometimes) are just natural when you share a work location. Based on the number of management seminars, consultants, books and boot camps on leadership, teamwork and communication out there, I have to think “comes natural” is probably overstated. Taking some care to understand how your coworkers feel connected, included, with contributions encouraged and welcomed is important to all teams, but people seldom do this. The result is worse then for remote workers who lack the physical interactions with all their cultural norms to at least partially help people get on the same page.

For the me the key things that are absolutely essential are these:

  • A group chat tool everybody is in all day. Currently I like skype for this
  • A group video conference tool with good screen sharing capabilities.  Currently I like gotomeeting and Google hangouts are too slow and two fuzzy. So is Skype for the most part which is too bad.
  • IM with presence actively maintained,
  • equal attention to IM conversations that you would give somebody walking up to your desk.
  • Daily “stand up” phone conversations to talk about progress, plans, open (but don’t solve) questions and problems
  • willingness to jump into short, focused phone calls, individually or in small groups when needed
  • A propensity to CC email to more rather than less people. 
  • inclusion in the non-work conversations, babies, cars, trips, bands, all the good stuff
  • face to face get togethers, more often at first as you get to know people, but at least once per month after. I personally like to meet once per week face to face
  • very limited restrictions on what you can actually do remotely. This means distributed source code systems, accessible systems, vpns, whatever it takes to be equally productive remotely. Don’t make the office work 100% better than the remote worker.

I find this works for me, and found it worked with remote teams in other countries as well, at least to the extent that time zones are an issue. If language is an issue, most people can read and write much better than they can speak and hear a second language, so emphasize IM and chat over the phone call.

Mostly, actively include the remote people and you will be fine. Anything else and you will be one of the many people that have failed to incorporate a distributed work force. And too bad for you!

Slides and sample from the Chippewa Valley Code Camp

Here are my slides and samples from the Chippewa Valley Code Camp. Read the readme for details of how to run the code. Thanks to all who attended, the few and proud come to mind!

Visual tracking of workflow is really a useful thing to build into your systems. The list of gotcha’s in the slides and others I’ve found since, mean that this particular approach using the rehosted workflow designer, while viable in certain cases, is probably going to take more work to make it generally usable solution for visual tracking of workflow. The idea presented here to use a custom tracking participant and SignalR to broadcast workflow tracking events is simple and solid and with a little more goo to pick a “topic” using groups or hubs would work very well I think.  A little care will need to be explored to account for scaling if you’d do this with hundreds of users based on the overall design of signalr.

I also would like to point out a few other related projects, one that used a service bus instead of signalr and another that used signalr, but not a wcf behavior or visual tracking. Enjoy your reading!—distributed-tracking-visualization.aspx