Tuesday, February 7, 2012

Upload Files to a Django form

Decided to add a little feature to the Remixerator the other day - the ability to supply your own base kickstart file. Django has a Filefield widget for its forms that make this work like magic. The form looks a bit like this.


class KickstartForm(forms.Form):
    select_language = forms.ChoiceField(choices=languages(), initial='en_US')
    select_timezone = forms.ChoiceField(choices=timezones())
    name_of_the_spin = forms.CharField()
   
    ks_choices = ((None, 'Use your own!'),) + ls_ks()
    based_on = forms.ChoiceField(choices=ks_choices)
   
    uploaded_kickstart = forms.FileField()
    # ... Set the widget css...



I then tossed in a little jquery to hide the form until the "Use your own!" option is selected...


// Show the "upload file" portion if the user decides to use their own kickstart file
$("#id_based_on").change(
  function() {
    if( $('#id_based_on option:selected').text() == 'Use your own!' ) {
      $("label[for='id_uploaded_kickstart']").show('fast');
      $('#id_uploaded_kickstart').show('fast');
    } else {
      $("label[for='id_uploaded_kickstart']").hide('fast');
      $('#id_uploaded_kickstart').hide('fast');
    }
  });


Next comes the handler for actually dealing with the file, which gets sent along with a POST


def handle_uploaded_ks(uploaded_ks):
    # MEDIA_ROOT defined with django settings, mine is set to a temp cache
    ks_path = os.path.join(settings.MEDIA_ROOT, uploaded_ks._name)
    print ks_path
    destination = open(ks_path, 'wb+')
    for chunk in uploaded_ks.chunks():
        destination.write(chunk)
    destination.close()


Then I needed to go to the view that handles the POST and make it execute the handler.

# The package view (page after the form)

def packages(request):
    """
    Select packages and groups
    """
    # if the user is uploading their own kickstart file, toss it in the cache directory
    if request.FILES:
        handle_uploaded_ks(request.FILES['uploaded_kickstart'])


And finally, we had to allow the form to handle all of this. This is simply done through the form tag's enctype in the html...
<form method="POST" action="/packages/" enctype="multipart/form-data">
    {{ form.as_p }}
    <input type="submit" value="Next" id="nextButton"/>
</form>




Tuesday, January 10, 2012

Python on Android with Kivy


So, it took some tinkering, but I managed to get python working on android using the python-for-android project that allows python kivy applications to run on an android device. I did stumble upon a few issues, though, so I'll describe my steps below.

Original Instructions can be found at the project's github page, and Mathieu Virbel's blog.

The general steps are making sure that you have all the SDK and NDK tools ready, recompiling python against Android (ARM processor), create a python distribution, and using that distribution to build an APK from your Kivy Application. Kivy is necessary to actually render an interface on the screen for user feedback, and overall it's a pretty promising multi platform UI framework. Other toolkits based on something other than OpenGL ES 2.0 can potentially work, should other java bootstraps get created in the future. As of right now, the android-for-python project is providing only a single java bootstrap to handle touch input and audio output, with support for more of android's features on the way.

Note: I used Ubuntu 11.10 to mimic the creator's environment.

Step 0: Get the SDK and NDK here:
http://developer.android.com/sdk/index.html
http://developer.android.com/sdk/ndk/index.html


Step 1:
After downloading this, run SDK program in android-sdk-linux/tools/android
Download the appropriate version of the API. For me, this was version 10 (running on 2.3)


Step 2:
Export necessary variables that are used in the configuration script
Make sure that the NDKVER and API numbers match those that you downloaded and installed.
export ANDROIDSDK="Location/android-sdk-linux"
export ANDROIDNDK="Location/android-ndk-r#"
export ANDROIDNDKVER=r#
export ANDROIDAPI=##


Step 3: get the minimal environment for building python
sudo apt-get install build-essential patch git-core cache


Step 4: clone the python-for-android project
git clone https://github.com/kivy/python-for-android.git


Step 5: Create the python distribution. This is where I first ran into problems.
ideally, you this will work by simply using the distribute.sh script
./distribute.sh -m "kivy package1 package2…"

(available packages are jpeg pil png sdl sqlite3 pygame kivy android libxml2 libxslt lxml ffmpeg openssl with more being created as the project moves forward, see the github repository's readme to contribute to this growing list)

However, here are some issues I ran across when trying to create a python distribution.
  - outdated awk tool. You may get an error message saying that AWK_HOME is referencing an old version of awk. Install a new version of awk, go into your android-ndk-r#/prebuilt/linux-x86-orx64/ directory and rename awk to awk_ so that the build script uses the global awk tool. If that doesn't work, have the AWK_HOME variable point to gawk instead.
  - missing NDK tools. The distribute script refers to the android-ndk-r#/platforms/android-##/ directory that refers to your selected version of the API denoted by the ANDROIDSDK export. However, your NDK might not contain contents for your selected version. For example, r7 of the NDK does not contain contents for version 10 of the API. This might be because its tools refer to the tools included in the existing android-9 directory. Either way, if your version of the API doesn't have a corresponding directory, create one, and copy the contents of the previous version into it.


Step 6: Create an APK using the build.py script in /python-for-android/dist/default (unless you specified a different location for the python distribution)

Package your APK using the following command:
python build.py --dir location_of_your_kivy_application --package your.package.name --name "My Application" --version # debug

If you are unfamiliar with kivy applications, you should check this out.
There are also android requirements for kivy applications:


Step 7: Deploy to android by executing the following command as root.
sudo location_of_sdk/platform-tools/adb install -r location_of_distribution/bin/app_name-version-debug.apk

Note that if it says you do not have correct permissions, you will need to restart adb and re-start it as root by doing the following commands:
adb kill-server
sudo adb start-server

Learn more about the adb tool here:
http://developer.android.com/guide/developing/tools/adb.html


So, with this success, I just need to think of something to create.

Saturday, December 24, 2011

Vacation Productivity

Not celebrating the holidays does give me something: free time during break.

I've been tinkering with a whole bunch of different things leisurely during my break thus far. Before leaving, I had attended RIT's STEM Challenge hackathon (didn't end up working on a STEM game, though) and decided with my buddy Logan Mitchell that we'd try porting a MUD engine I had written some time ago in Java to Python, and make it better. It would be a good side-project to learn some things about the language.

One of the notable cool things I picked up was a nice way of generating the common "help files" for commands in the MUD, in a way that allows allowed us to document these commands at the same time. I had an idea. What if we could use the function docstring, and through reflection, generate a helpfile from it? Turns out, python can make that happen pretty easily. Inside our commands.py module...


import sys
import inspect
...

def help(args):
    """
    Check the helpfiles for something.
    syntax: help <subject>
            subject - the subject or command which you want to check for help on
    """
    ...
    # args is a namedtuple
    helpFunctionName = args.tokens[1]
    # use reflection to get a list of this module's function object
    functions = inspect.getmembers(sys.modules[__name__], inspect.isfunction)
   
    for functionName, function in functions:
        if functionName == helpFunctionName:
            docstring = function.__doc__
   
    prelude = "help file for: " + args.tokens[1] + "\n"
    dashline = ("-"*len(prelude))
    return prelude + dashline + docstring

So, calling "help help" in our MUD will actually pull out help's docstring and have it contribute to the help file. Pretty cool stuff.



Monday, December 12, 2011

Dorrie part II

I did some more stuff with Dorrie last week.

Beyond some aesthetics changes, I improved the usability of Dorrie through tooltips and previews. This led to a bunch of fun with the well-known jQuery Tools tooltip library. Using the library to create nice looking tooltips was pretty simple, once things got set up. It just boils down importing the library (and make sure that you import it after the general jQuery library), and setting up the function call inside the document ready clause like so:

$("match").tooltip();

This will call the default tooltip aesthetic, pulling in the text from the matched element's title field. Of course, this is the easiest scenario. Getting things just right to match the aesthetics of the site required a lot of tweaking.

On the home page of Dorrie, I list the steps that the process will take you through for creating your remix. After some feedback, I was told that these steps should have their terms defined, so this was a pretty good place to start for adding these tooltips. The code ended up looking like this:

$("#steps a[title]").tooltip({ effect: 'fade', tipClass: 'tooltipBig'});

The jQuery library allows the user to define multiple parameters for these tooltips. The most important, I found, were the effect, tipClass, and position parameters. The effect parameter defines the particular animation the tooltip uses upon popup (There are default styles, which are quite nice, but the library supports the ability to create custom animations, too). The tipClass parameter points to the css that the tooltip will use (allowing you to have multiple styles and not rely on the "tooltip" css class). Position defines the position where the tooltip will pop up.

In my fork of the Dorrie project, I added a few different styles of tooltips. Each one had their own respective bubble images that needed adjusting (all of mine derive from the free-to-use minimalist images jQuery tools provides), and positioning that needed tweaking.

One of the more interesting areas of using these tooltips is in the /packages/ step. Here, I add tooltips for package categories by pulling in the data that Dorrie uses in the backend making use of Django's Mako support:

<a href="#" title="{{cat.description}}">{{ cat.name }}</a>

These <a> tags make up categories that the jQuery accordion widget uses. The title fields are pulled in from the python backend (packages.py) data, a collection of python objects that represent categories, groups, and individual packages. This data is initialized on service startup. This data is in turn pulled in from Fedora's huge comps file that defines the packages it has. Overall, pretty nifty. Unfortunately, the way Dorrie currently pulls in the list of package groups within a category is different from the way it handles categories. Instead of bringing in a list of package groups that use the backend group model, it's just a list of strings. I'm not sure why it's done this way, and I intend to change it soon.

I also made an attempt to take control of jQuery Tools tooltip's way of embedding HTML in a tooltip so that I could create some nice tooltip image previews of the desktop environments. Using mako's loops, the html for each group was displayed, and inside of that it was easy to create a simple condition for adding in the tooltip...




{% for cat in cats %}
<h4><a href="#" title="{{cat.description}}">{{ cat.name }}</a></h4>
<div>
{% for group in cat.groups %}
<input type="checkbox" id="grp_{{ group }}" value="{{ group }}"
onClick='someOnclickFunction("{{ group }}")' />

{% ifequal cat.name "Desktop Environments" %}
Add CSS class for tooltip matching
Create HTML div for a tooltip
{% else %}
Default Behavior
{% endifequal %}

<br />
{% endfor %}
</div>
{% endfor %}


There were some problems, however. The html loaded inside the tooltip is by default the next section. This led us with two problems.

If we defined the HTML inside the loop, it would be generated multiple times - once per group - and could generate content based on the backend model, with data pulled in using mako. This would be fine, except that Mako generates the html before the jQuery library checks for the html to snag for the tooltip. This results in the first div being pulled out for the tooltip, and the rest displaying outside any tooltip. Not what we wanted. If I were to pull out the tooltip div from the loop, and create an explicit reference to it, the content would not be dynamic, as it would not have been generated within the loop.

I quickly looked for an alternative solution for a simple tooltip image preview, and stumbled across Alen Grakalic's script over at CSSGlobe, which does the trick pretty much flawless. I used his code as a base and am adjusting it to work for my solution (I have to finish and make the image position dynamic based on the page, not mouse position).

I'll be sure to post again when I get some of these things working, to detail other changes I've made.

Wednesday, November 9, 2011

Dorrie

So after abandoning my misadventures of last week, I decided to clone the Dorrie project, and use that as the underlying base for the remixerator. The bread and butter of this project is in the pykickstart.parser module, which has many tools to parsing and generating a working kickstart file. I will briefly explain the key functions and objects that are used in dorrie.

pykickstart.parser defines three classes:
KickstartParser - a state machine used when going through a kickstart
Packages - an object that encapsulates all the selected (and removed) packages of the kickstart.
Script - an object that encapsulates a script of the kickstart file.



The KickstartParser's most useful function for us is the readKickstart() function. This, given a kickstart file, will go through (using the state machine internally) and create all the objects necessary to represent a particular kickstart. This uses internal parsing and handling functions. It also has an internal handler object that saves and executes commands that are parsed.

The Packages object has four important lists. The lists excludedList and packageList, that contain the actual packages for the kickstart (the former being denoted with a '-' character). The lists groupList and excludedGroupList are also included in this module. Essentially, when we want to add or remove any particular package, we add the package name to the respective lists.

The Script module has two important fields: script and type. Each script object should encapsulate only one script, the script field set to the actual script's string, and the type set to the relevant type found in pykickstart.constants. Creating a script object is not difficult, however, adding it in, is. Pykickstart has a BaseHandler object (to which KickstartParser's handler object is a subclass), that contains a scripts section, holding these instances of the script object.

One of the difficulties in using the dorrie project for the remixerator is that adding a post script is non-trivial. The documentation states that to add a script, you call KickstartParser.addScript... but this function actually doesn't exist. For the time being, one way to get around this is to literally append %post script to the returned formatted string. I did this for the sake of time, but this will be changed when I can find a more proper way to add a script object.

After this, I had to go through Dorrie's settings.py file and change references to reflect fedora 16. This included versions of the internal parser (these are defined in the latest pykickstart repository), and package trees.

Next is a little bit of de-chroming, to get the aesthetics looking like an RIT project.

Edit: Scripts get added when the Script "section" object has its finalize method called. I'll post more on this later.


Thursday, November 3, 2011

HTML and JQuery forms on Twisted

When I started to prototype a remixerator, I created a simple form front-end using simple html and JQuery.

Here's the head. I used this nice css theme generator to get an aesthetic that worked, put in a few of my own tweaks, and imported the necessary JQuery files. In my particular form, I liked the idea of having "sections" as tabs, and "subsections" represented by the accordion layout. When using accordions inside tabs, make sure to bring in accordions FIRST and tabs SECOND or else they won't render correctly.



<head>
  <title>Remixerator Prototype</title>
  <link type="text/css" href="css/custom-theme/jquery-ui-1.8.16.custom.css" rel="stylesheet" />
  <link type="text/css" href="css/overall.css" rel="stylesheet"/>
  <script type="text/javascript" src="js/jquery-1.6.2.min.js"></script>    
  <script type="text/javascript" src="js/jquery-ui-1.8.16.custom.min.js"></script>      
  <script type="text/javascript">
  $(function(){
    // Accordion
    $("#accordion").accordion({ header: "h3" });
    // Tabs
    $('#tabs').tabs();

    $("#RemixPostForm").submit(function(e){
      console.log("Entering Submit");
      $.post('http://localhost:8080/handler.html', function(data) {
        console.log("Entering Submit Callback");
      });
    });
  });
  </script>
</head>


Now for the body. Tabs are represented as a list with ids, and each tab's content is a div with a matching id. Otherwise, this is pretty standard html form stuff. I spent [too much] of my time dealing with css issues with this form.


<body>
  <form id="RemixPostForm">
  <!-- Tabs -->
  <h2 class="demoHeaders">Remixerator</h2>
  <div id="tabs">
    <ul>
      <li><a href="#tabs-wall">Wallpaper</a></li>
      <li><a href="#tabs-apps">Applications</a></li>
      <li><a href="#tabs-create">Create</a></li>
    </ul>

    <div id="tabs-wall">
      <h3>First, select your wallpaper.</h3>
      <div id="wallpaper_table" align = "center">
        <table id="imagegrid" border="0" width="400">
          <tr>
            <td align="center"><img src="images/ritwallpaper.jpg" alt="gallery thumbnail" height = "80" width = "120"/></td>
            <td align="center"><img src="images/f15wallpaper.png" alt="gallery thumbnail" height = "80" width = "120"/></td>
          </tr>
          <tr>
            <td align="center"><input type="radio" name="ritwallpaper_cb" id="ritwallpaper_cb" checked="checked"/></td>
            <td align="center"><input type="radio" name="f15wallpaper_cb" id="f15wallpaper_cb"/></td>
          </tr>
        </table>
      </div>
    </div>

    <div id="tabs-apps">
      <h3>Next, choose the applications you want.</h3>
      <div id="accordion">
        <div>
          <h3><a href="#">Office and Text</a></h3>
          <div>
            <input type="checkbox" name="vim_cb" id="vim_cb"/>Vim<br>
            <input type="checkbox" name="emacs_cb" id="emacs_cb"/>Emacs<br>
            <input type="checkbox" name="nano_cb" id="nano_cb"/>Nano<br>
            <input type="checkbox" name="libre_cb" id="libre_cb"/>Libreoffice<br>
          </div>
        </div>
        <div>
          <h3><a href="#">Video, Images, and Music</a></h3>
          <div>
            <input type="checkbox" name="inkscape_cb" id="inkscape_cb"/>Inkscape<br>
            <input type="checkbox" name="gimp_cb" id="gimp_cb"/>Gimp<br>
            <input type="checkbox" name="dia_cb" id="dia_cb"/>Dia<br>
            <input type="checkbox" name="banshee_cb" id="banshee_cb"/>Banshee<br>
          </div>
        </div>
        <div>
          <h3><a href="#">Communication and Internet</a></h3>
          <div>
            <input type="checkbox" name="firefox_cb" id="firefox_cb"/>Firefox<br>
            <input type="checkbox" name="elinks_cb" id="elinks_cb"/>ELinks<br>
            <input type="checkbox" name="midori_cb" id="midori_cb"/>Midori<br>
            <input type="checkbox" name="pidgin_cb" id="pidgin_cb"/>Pidgin<br>
          </div>
        </div>
        <div>
          <h3><a href="#">Utility</a></h3>
          <div>
            <input type="checkbox" name="zsh_cb" id="zsh_cb"/>Z-Shell<br>
            <input type="checkbox" name="git_cb" id="git_cb"/>Git<br>
            <input type="checkbox" name="htop_cb" id="htop_cb"/>Htop<br>
            <input type="checkbox" name="powertop_cb" id="powertop_cb"/>Powertop<br>
            <input type="checkbox" name="screen_cb" id="screen_cb"/>Screen<br>
            <input type="checkbox" name="wine_cb" id="wine_cb"/>Wine<br>
          </div>
        </div>
        <div>
          <h3><a href="#">Fun and Games</a></h3>
          <div>
            <input type="checkbox" name="nethack_cb" id="nethack_cb"/>Nethack<br>
          </div>
        </div>
      </div>
    </div>
    <div id="tabs-create">
      <h3>Now click the button below to start making your remix.</h3>
      <input type="submit" value="Go!" name="start_button" id="start_button" />
    </div>
  </div>
</form>
</body>


Lastly, the twisted bits. I simply created a simple service, server.py, which hosted the form, and had the handler.


from twisted.web import static, http, resource, server
from twisted.web.server import Site
from twisted.web.static import File
from twisted.internet import reactor
from twisted.web.resource import Resource
import cgi
import sys

class FormPage(Resource):
  isLeaf = True
  allowedMethods = ('GET', 'POST')
  def __init__(self):
    resource.Resource.__init__(self)
    def render_GET(self, request):
      return self.render_POST(request)
      def render_POST(self, request):
        #Modify Kickstart, do validation
        return '<html><body>ISO Created!</body></html>'

if __name__ == "__main__":
  root = static.File('/Users/eitanromanoff/Documents/github/Remixerator')
  indexPage = Resource()
  formHandler = FormPage()
  root.putChild('index.html', indexPage)
  root.putChild('handler.html', formHandler)
  reactor.listenTCP(8080, server.Site(root))
  reactor.run()



The Remixerator

I've been working on a new project as of late that uses the kickstart tinkering I did at the start of the quarter, and takes it a step further. The goal is to create a "remixerator" - an interface, ideally a web-based one, that allows a user to create a custom fedora remix (that is still RIT-themed in content and aesthetics). The end goal would be to put it into a machine similar to the kiosk, and have a very controlled and automated process for taking in a DVD, customizing the remix, and then burning the image for a user.

The customization of the remix is easy enough - it's just a generation of a kickstart. All of my custom scripting that was done can still be injected into the kickstart, what really changes is the configuration of applications. To start, I looked into various existing technologies that do similar things.

LiveUSB creator was not really relevant to what I was trying to do.
SuseStudio was very similar, but based on several cloud technologies that are not necessary to my project at hand. Furthermore, our RIT remixes were to run Fedora.
Kickstarter was too much targeted towards Meego.

And there were several other applications as well.

While I was  no eager to re-invent the wheel, none of the above projects really fit the bill. Ultimately, I decided that all I would have to do is make a web form (perhaps take the opportunity to learn some JQuery), generate a kickstart on submit, run the creation process, and serve up a webpage when it was finished.

The form was easy enough, just a bunch of html and JQuery running on twisted. Hosting things on twisted was pretty simple, too, to get started. However, things quickly became less trivial. Launching the script on a post was simple, but being able to constantly update a page, given the output of the create livecd process, would be difficult to tackle, and it was hard to believe there was another tool available that did some of this for me.

Looking again the other day, I stumbled across the dorrie project, which does much of what I wanted to do. It also has a convenient spot in its parser where I could likely inject the postscript sections that I wrote for the RIT remix. I'll take this approach from here on out, and bother with aesthetics later.