lundi 25 juillet 2016

AbstractMethodError when using traits in Groovy/Grails

Introduction

The 2.3 version of groovy introduced traits, sets of fields and methods that can be implemented by classes like superpowered interfaces or abstract classes; they not only contain the declaration, but also the definition of methods and can contain attributes. A class can implement multiple traits at once, providing some kind of multiple inheritance for groovy that classes could not provide in java.

AbstractMethodError

This feature being quite new, the compiler still has some glitches when dealing with traits. The most notable error is that when you change the source of a trait, it won't recompile the classes implementing it, causing sometimes an AbstractMethodError at runtime when the class doesn't find a recently added or modified method of the trait.

Resolution

To force the recompilation of the sources files, you can either do a full clean and rebuild of your project - but it can be a huge loss of time for a large project - or add a trivial modification to each source file implementing the trait to trigger their recompilation instead of using the cache; adding a comment for example, then compile - but for a trait that is widely used, like an helper trait used in every unit test, adding a comment to a large number of file can be tedious, and you must remember to remove them before committing...

A better solution

A better solution is to use a script add these comments in all the files of a folder to trigger their rebuilding and another one to remove the comments. This way you avoid the manual modification of the files while still avoiding the rebuild the full project.

The workflow after getting an AbstractMethodError:

  1. Run the mark_files_as_changed.py script on the source folder that contains the classes implementing the trait. For a test helper trait use it on the test/ folder, for a controller helper trait run it on the controllers/ folder.
  2. Recompile you project. Only the 'changed' classes, that have been marked by your script will be recompiled.
  3. Run the unmark_files.py on the same folder, removing the comments added by the first script  to clean the files.

Implementation in Python:

mark_files_as_changed.py:

"""
This script appends comments to all files in the folder whose path is given as parameter, to ensure that
Intellij IDEA rebuilds them. This is useful to ensure that classes that implements a trait are recompiled when that trait is edited.
After compilation, please run the unmark files to remove the added comments.
Created on Tue Jul 19 14:04:51 2016
@author: hschoonjans
Python 3
"""

import os
import sys

walk_dir = sys.argv[1]

print('walk_dir = ' + walk_dir)

print('walk_dir (absolute) = ' + os.path.abspath(walk_dir))

APPEND = 'a'

for root, subdirs, files in os.walk(walk_dir):
    print('--\nroot = ' + root)

    for filename in files:
        file_path = os.path.join(root, filename)

        print('\t- file %s (full path: %s)' % (filename, file_path))

        with open(file_path, APPEND) as file:
            file.write("//CHANGED")


unmark_files.py:


"""
See mark_file_as_changed.py.
Removes the "//CHANGED" comments in all the files of the folder.
Created on Tue Jul 19 14:04:51 2016
@author: hschoonjans
Python 3
"""

import os
import sys

walk_dir = sys.argv[1]

print('walk_dir = ' + walk_dir)

READ_MODE = 'r'
WRITE_MODE = 'w'

for root, subdirs, files in os.walk(walk_dir):
    print('--\nroot = ' + root)

    for filename in files:
        file_path = os.path.join(root, filename)

        print('\t- file %s (full path: %s)' % (filename, file_path))

        f = open(file_path, READ_MODE)
        file_data = f.read()
        f.close()

        new_data = file_data.replace("//CHANGED", "")

        f = open(file_path, WRITE_MODE)
        f.write(new_data)
        f.close()


You can also find the implementation on my GitHub.


jeudi 14 juillet 2016

Practical guide to neural style with docker

Google has recently made some noises with it's amazing neural dream images, images generated by neural networks. Another popular use of neural networks for image processing is called "neural style", the use of neural algorithms to apply the style of an image to another.

I'll explain here how to quickly use neural style algorithms to transform your photos. I'll assume you aren't interested into the theory behind the method and just want to get started quickly. To avoid a cumbersome installation and possible incompatibilities of the technologies used with your machine, I'll make you use a virtual machine called Docker and download an image with all the needed libraries already installed.

First, you'll need to have Docker installed on your computer. I'll assume that you will manage to do it by youself.

With docker installed, pull the image containing the code:
docker pull heschoon/neural-style

It will take some time, as you are downloading a 1Gb virtual machine. While it downloads, collect at least two images in one folder on your computer. We will together learn how to apply the style of the first image ( let's call it style.png ) to the second image ( content.png ).

Tip: choose as style.jpg an image with a memorable style!

A popular style image is the starry night painting of Van Gogh:


In this tutorial, I'll use the picture of an handsome programmer as content.jpg.


Put both image in the same folder on your computer. Now you can run the virtual machine:
docker run -it -v <path/to/your/images/folder>:/root/input-images/ \
heschoon/neural-style /bin/bash

In case you wonder what' does the command: "docker run" will run the virtual machine, "-it" attaches an interactive terminal on your virtual machine, "-v <path/to/your/images/folder>input-images" mounts the folder containing the images on the virtual machine, "heschoon/neural-style" is the name of the virtual machine to run. This docker image is based on the jcjohnson/neural-style implementation of the neural-style algorihtm.

You can now move to the folder that contains the code:
cd ~/neural-style/

Ensure that the image folder has been successfully mounted on your docker:
ls ../input-images

If the command list the names of the images you have in your folder; probably style.jpg and content.jpg, the folder has been successfully mounted.

You can generate a new image:
th neural_style.lua -gpu -1 -style_image ../input-images/style.jpg \
-content_image ../input-images/content.jpg -image_size 256

The algorithm is now running. It my laptop it took 2 hours, so be patient.You might want to give more RAM to Docker if the algorithms crashes.

When the wait is over, copy the generated images into the shared folder, to make them persistent and accessible from your computer:
cp out* ../input-images

You can now exit the virtual machine by type the exit command, and look into your image folder. The generated output images should be available.



Now, you can try with other style and content images, and play with the other flags and commands described on the code repository.


lundi 6 juin 2016

Controlling the mouse with the keyboard

Most programmers prefer to using the keyboard to the mouse. While many programs and web browsers can be used with the keyboard only, some don't.

If you are frustrated of using the mouse for those archaic tools, you might be glad to learn that there exists some settings on Windows that, enabled, makes possible the control of the mouse with the keyboard.

This is a built-in feature in Windows 7.

mercredi 18 mai 2016

java.lang.ClassNotFoundException: org.springsource.loaded.ri.ReflectiveInterceptor

I was stuck today on some unhelpful error message when trying to run my Grails app on Intellij IDEA 15.0.6:
Exception in thread "main" java.lang.reflect.InvocationTargetException
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:606)
    at org.springsource.loaded.ri.ReflectiveInterceptor.jlrMethodInvoke(ReflectiveInterceptor.java:1426)
    at com.intellij.rt.execution.CommandLineWrapper.main(CommandLineWrapper.java:46)
Caused by: java.lang.NoClassDefFoundError: org/springsource/loaded/ri/ReflectiveInterceptor
    at com.intellij.rt.execution.application.AppMain.main(AppMain.java:125)
    ... 6 more
Caused by: java.lang.ClassNotFoundException: org.springsource.loaded.ri.ReflectiveInterceptor
    at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
    at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
    at java.security.AccessController.doPrivileged(Native Method)
    at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
    ... 7 more

This Stack overflow answer solved my issue: I had to go into the .idea folder located in the root of my project, open the workspace.xml file and set the dynamic.classpath option to false.

samedi 30 janvier 2016

Aliases of aliases: myth or reality?

The dangers of aliases of aliases in Elasticsearch

This post introduces the Elastic aliases and explain the danger of misusing them.

The aliases

In Elasticsearch, aliases provide a way to give alternative names to an index or a group of index. Searching, deleting and adding documents over multiple indexes as if manipulating only one index is made easy with this technique.

Aliases can also be used in your application code instead of the index name to provide the possibility of changing the index without modifying the source and to make zero-downtime reindexing. With the addition of filters, aliases can give the impression different small communities each have their own index, by filtering the retrieved documents by some user identifier, while in fact using a single index.

Aliases of aliases

Going further, you could imagine many use cases where aliases of aliases could provide a way group more aliases and add additional filters to them. As an example, in the shared index case, you could want an alias of alias to refer to three specific communities and add a filter to match only the documents beyond a specific date.

Does it work? Let's try it! Launch a local Elastic instance and run the following queries in your sense:

 DELETE index
 POST index
 {
     "mappings": {
         "user": {
             "properties": {
                 "name": {
                     "type": "string"
                 }
             }
         }
     }
 }
 
 POST _aliases
 {
     "actions": [
        {
           "add": {
              "index": "index",
              "alias": "alias1",
              "filter" : { "term" : { "name" : "helain" } }
           }
        },
        {
           "add": {
              "index": "index",
              "alias": "alias2",
              "filter" : { "term" : { "name" : "nialeh" } }
           }
        }
     ]
 }

 POST _aliases
 {
     "actions": [
        {
           "add": {
              "index": ["alias1","alias2"],
              "alias": "meta-alias"
           }
        }
     ]
 }


With these steps, you just created the index and two aliases alias1 and alias2 of this index, each alias with it's own filter. The last operation added an alias meta-alias covering alias1 and alias2.

Let's now see our aliases with a GET _aliases:

{
   "index": {
      "aliases": {
         "meta-alias": {},
         "alias2": {
            "filter": {
               "term": {
                  "name": "nialeh"
               }
            }
         },
         "alias1": {
            "filter": {
               "term": {
                  "name": "helain"
               }
            }
         }
      }
   }
}

From the result, we can see that our meta-alias doesn't refer to both our aliases, but only to our index. If you index some documents and try to search on meta-alias, you will see that the filters of alias1 and alias2 aren't applied to your queries.

This behavior happens because aliases aren't real indexes; a query on an alias will be forwarded to the referred index, with the addition of the filter if existing. When we created meta-alias, an alias creation query was sent to alias1 and alias2, that forwarded it to index. Meta-alias consequently forwards it's requests to index only and the filters of alias1 and alias2 will never be applied. If you don't expect  this behavior, it can results into exposure of information you would have wanted filtered.

Conclusion

Creating an alias of an alias has the same result as if creating an alias over the referred indexes directly.

lundi 11 janvier 2016

Elasticsearch in 20 minutes

Want to setup an Elasticsearch instance, tinker with this technology? Here is a quick and dirty tutorial that will help you to setup an Elasticsearch instance on your computer in less than twenty minutes!

What's Elasticsearch?

If you have an idea of what's Elasticsearch is you can pass this section and go straight to the installation of Elastic.

Elasticsearch ( also called Elastic ) is a database based on Lucene. It's becoming extremely popular since it's free, open-source and makes searches and data analytic easy and in real-time.

While a SQL query might take minutes or event timeout when the database contains too much records, Elasticsearch will scale better and always return a result in milliseconds. I have once tried to empirically compare the performance of both technologies, made a lot of equivalent queries on hundreds of thousands documents on both database. Elastic was overall 3000 faster, but also provided advanced lots of Natural Language Processing features like tokenization, use of n-grams, stop-word removal and many more, making possible the implementation of a google-like full-text search engine on your websites, making search more natural for your visitors.

Installing Elasticsearch

To start you journey with Elasticsearch, just download the software from the official website. When the download is over, just extract the contents of you elasticsearch-x.x.x.zip and go into the resulting folder.

Open a command prompt in the folder and move into the bin directory. Run the elasticsearch script to launch the database.


Your first Elasticsearch instance is now running! In the next tutorials, we will see how to add documents into your Elastic server and unlock the power of real-time full-text searches and analytics.


  • Tip: if you get a "Could not find any executable java binary." or "JAVA_HOME environment variable must be set" error, you should install Java and edit your environment variables.